feat(qwen38): score the inherited corpus at the window the container really has

The reference rig declared a 32,768-token window while the boot ledger hands it
7,436, so 197 of the 198 inherited documents were too long to score. The engine
warns about the real pool; the flag was the only cited source.

Add scripts/rechunk-kld-corpus.py, which moves document boundaries without
moving tokens: each record is re-cut into windows of at most --max-doc-tokens
(7,168 = 7x1024, the largest whole kilopage under 7,436), and each window gets
its own score_from = len - --score-tail. The reference scores the same token
series at more positions and at a shorter context; the calibration corpus is
untouched, so no quant moves. Writes go through a temp file and the token
counts must match before the output is published.

The rig header now says what the pool actually is, and scripts/test-changed.sh
routes the new script to its own seam instead of the generic syntax fallback.

AGENTS.md

@@ -166,19 +166,22 @@ bare compose container never returned; 2026-09-11: five observability
 services had no units). The observability services now have units under
 `dankclaw/observability/quadlet/` and their compose definitions were removed —
 one owner per container name. The stack is **configured and delivering as of
-2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are
+2026-09-17**: Grafana datasources (Prometheus and Loki) and the `llm-fleet`
-provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
+dashboard are provisioned from `shared/grafana/provisioning/`, and
-llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
+`shared/prometheus/rules/llm-fleet.yml` loads two rule groups — `llm-fleet`
+(llama.cpp/halogen, `llamacpp:*`) and `vllm-fleet` (radiance, `vllm:*`) — alert
++ recording rules, and firing rules are
 delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
 alertmanager.container`, loopback :9093) → host-side Signal relay
 (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
 IP, the one bind a container can reach as `host.containers.internal`;
 iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
-Prometheus targets read down for structural reasons (HTML-scraped "probes",
+The former structural target downs (HTML-scraped "probes", Gitea metrics off)
-Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
+are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
-a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
+(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
-without `--metrics`. See
+every Prometheus job reports up. `frank-zealot` was down for a different reason
-`serverAppConfigs-docs/Infrastructure/Observability.md`.
+until 2026-10-18: llama.cpp will not serve `/metrics` at all without
+`--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
 
 ## Testing
 
@@ -417,16 +420,17 @@ disagrees with it is the bug.
   digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
   Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
   `/home/dank/logs/frank_dragoon.log`, NOT the journal.
-- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
+- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
-  is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
+  **per-request ceiling** (a longer prompt is a 400, never a truncation);
-  `HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
+  `HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
-  the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
+  pool positions buy concurrency, the per-request window never moves;
-  drafter always armed (the adaptive default disarms it and halves decode);
+  `HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
-  server sampler 1.0/0.95/20.
+  default disarms it and halves decode); server sampler 1.0/0.95/20.
-- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
+- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
-  The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
+  ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
-  under-counts, so a harness window equal to the ceiling makes every long
+  estimate under-counts (trigger is 95 %), so a harness window at the ceiling
-  request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
+  makes every long request a guaranteed 400.
+  Reasoning arrives in `reasoning_content`; the
   template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
   OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
   `/completion`, `/ping`, `/props`, `/tokenize` are 404.

dankclaw/authentik/quadlet/authentik-outpost.container

@@ -20,7 +20,7 @@ Wants=authentik-server.service
 
 [Container]
 ContainerName=authentik-outpost
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/authentik/quadlet/authentik-postgres.container

@@ -14,7 +14,7 @@ After=network.target
 
 [Container]
 ContainerName=authentik-postgres
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
 Environment=POSTGRES_DB=authentik

dankclaw/authentik/quadlet/authentik-server.container

@@ -19,7 +19,7 @@ Wants=authentik-postgres.service
 
 [Container]
 ContainerName=authentik-server
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
 Exec=server
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/authentik/quadlet/authentik-worker.container

@@ -16,7 +16,7 @@ Wants=authentik-server.service
 
 [Container]
 ContainerName=authentik-worker
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env

dankclaw/gitea/quadlet/gitea-postgres.container

@@ -15,7 +15,7 @@ After=network.target
 [Container]
 ContainerName=gitea-postgres
 Network=gitea.network
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=POSTGRES_USER=gitea
 Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}

dankclaw/gitea/quadlet/gitea-server.container

@@ -19,7 +19,7 @@ After=gitea-postgres.service network.target
 [Container]
 ContainerName=gitea-server
 Network=gitea.network
-Image=docker.io/gitea/gitea:latest
+Image=docker.io/gitea/gitea:latest@sha256:30777c91280d973fa02be48b1ade735bd6859e8f6f7af3b4444d519437852504
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=DATABASE_TYPE=postgres
 Environment=DATABASE_HOST=gitea-postgres
@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
 Environment=DATABASE_PORT=5432
 Environment=USER_UID=1000
 Environment=USER_GID=1000
+# Prometheus scrapes /metrics over the published LAN port (job "gitea" in
+# shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
+# permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
+# the container; the only route in is this host's published 3000, and the
+# exposed surface is counters only — same posture as the LLM /metrics.
+Environment=GITEA__metrics__ENABLED=true
+Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
 PublishPort=0.0.0.0:3000:3000
 PublishPort=2222:22
 Volume=/home/dank/Containers/gitea/data:/data

dankclaw/marreta/quadlet/marreta-browser.container

@@ -23,7 +23,7 @@ After=gluetun.service network.target
 [Container]
 ContainerName=marreta_browser
 Network=gluetun.container
-Image=docker.io/lightpanda/browser:nightly
+Image=docker.io/lightpanda/browser:nightly@sha256:979c36ed869f1016b8dcf22939a0633c262f2c30e892a03a9f720eba2ff1931e
 
 [Service]
 TimeoutStartSec=300

dankclaw/marreta/quadlet/marreta.container

@@ -32,7 +32,7 @@ After=gluetun.service marreta-browser.service network.target
 [Container]
 ContainerName=marreta
 Network=gluetun.container
-Image=ghcr.io/manualdousuario/marreta:latest
+Image=ghcr.io/manualdousuario/marreta:latest@sha256:30f2a34567b79b66da794b4b9423751bbbcd07ceff600d9f6a2d219e7e439362
 # Relocate off SearXNG's 8080 inside the shared netns.
 Environment=NGINX_HTTP_PORT=8085
 Environment=APP_NAME=Marreta

dankclaw/observability/quadlet/alertmanager.container

@@ -33,7 +33,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=alertmanager
-Image=docker.io/prom/alertmanager:latest
+Image=docker.io/prom/alertmanager:latest@sha256:e9733bafb1bdef9b00e25a21f8f99dc26a22224bf16641ad754d1649f4c3357a
 Network=dankclaw_default
 User=0:0
 Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager

dankclaw/observability/quadlet/alloy.container

@@ -1,55 +1,51 @@
-# Quadlet unit: alloy (rootless, boot-persistent).
+# Quadlet unit: alloy (ROOT system service, boot-persistent).
 #
-# Install path on dankclaw: ~/.config/containers/systemd/alloy.container
+# Install path on dankclaw: /etc/containers/systemd/alloy.container
-# Unit name: alloy.service (container name: alloy)
+# Unit name: alloy.service (container name: alloy) — SYSTEM manager, root.
 #
-# Log shipper. This replaces the compose service of the same name in
+# WHY ROOT (not rootless, unlike its observability siblings). Alloy reads the
-# dankclaw/compose.yaml ("Observability" block).
+# host journal. The journal files are root:systemd-journal 0640 (other::---);
+# a rootless container carries no supplementary groups and cannot open them —
+# a rootless read of system.journal returns DENIED (see the Alloy page in the
+# vault). Running as root, one loki.source.journal reads system.journal AND
+# user-1000.journal, so it carries both host logs and the rootless Quadlet
+# containers' logs. Validated 2026-10-20: a root probe read 31,825 journal
+# lines and pushed 26,966 into Loki in 20 s, zero drops.
 #
-# IMAGE. franksdank/alloy is built locally by deploy-remote.sh
+# NETWORK=host. Loki publishes 127.0.0.1:3100 (loopback only) on the host, and
-# (build_compose_image) from dankclaw/alloy/Containerfile: upstream
+# no bridge container — rootless or root — can reach the host's loopback, so
-# grafana/alloy plus a static busybox for the healthcheck. Pull=never keeps a
+# this unit runs on the host network and pushes to 127.0.0.1:3100. The side
-# boot from reaching for that local tag at a registry.
+# effect: the UI must bind loopback too, so Exec sets the listen-addr to
+# 127.0.0.1 (the old rootless unit used 0.0.0.0 only because container loopback
+# was not host loopback; under Network=host they are the same address, so the
+# unauthenticated UI stays loopback-only — nothing off-host reaches it).
 #
-# NETWORK. It joins dankclaw_default so its config's loki.write endpoint
+# THE /etc/machine-id MOUNT IS LOAD-BEARING. sd_journal locates the journal
-# resolves the container name loki:3100.
+# directory by the local machine id; without it the container opens nothing and
+# the journal source reads exactly 0 lines — silently, no error, health green.
+# This mount is the entire reason the pipeline carries data. Never remove it.
 #
-# THE UI PORT EXISTS ON PURPOSE. Alloy's UI listens on the container's own
+# IMAGE. Built rootless by build_compose_image alloy, then mirrored into the
-# 127.0.0.1 by default and cannot be moved from the config file (alloy 1.x
+# root image store by the deployer (a root build runs in the build container's
-# refuses the ui block), so the compose service had no reachable health
+# own resolver and fails on DNS here). Pull=never: boot must not reach for the
-# endpoint at all: the homepage status card for alloy reported "down"
+# local tag at a registry.
-# continuously and there was nothing for a deploy to wait on. The command
-# below therefore moves the UI to the container's wildcard address and the
-# port is published on host loopback only, which puts alloy in the same class
-# as the other three: reachable from the host, by Caddy over the private
-# network, and by nothing off-host. Alloy has no authentication, which is
-# exactly why it is not published wider.
-#
-# SOURCES. The host journald socket and the host log/container-storage trees
-# are bind-mounted read-only; Alloy tails them and pushes to Loki. The config
-# lives at /opt/alloy/config/config.alloy, installed from
-# shared/alloy/config.alloy on every deploy.
-#
-# ORDERING. Wants/After loki.service: without a reachable Loki, Alloy's write
-# client fails and its log pipeline backs up.
 
 [Unit]
 Description=Alloy log shipper (Dankclaw)
-After=network-online.target loki.service
+After=network-online.target
-Wants=network-online.target loki.service
 
 [Container]
 ContainerName=alloy
 Image=localhost/franksdank/alloy:latest
 Pull=never
-Network=dankclaw_default
+Network=host
-Exec=run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
+Exec=run --server.http.listen-addr=127.0.0.1:12345 /etc/alloy/config.alloy
-PublishPort=127.0.0.1:12345:12345
+Volume=/opt/alloy/config:/etc/alloy:ro
-Volume=/opt/alloy/config:/etc/alloy
+Volume=/etc/machine-id:/etc/machine-id:ro
 Volume=/run/systemd/journal/socket:/run/systemd/journal/socket:ro
-Volume=/var/log:/var/log:ro
+Volume=/var/log/journal:/var/log/journal:ro
-Volume=/var/lib/containers:/var/lib/containers:ro
+Volume=/run/log/journal:/run/log/journal:ro
-HealthCmd=/usr/local/bin/busybox wget --spider -q http://localhost:12345/-/healthy
+HealthCmd=/usr/local/bin/busybox wget --spider -q http://127.0.0.1:12345/-/healthy
 HealthInterval=30s
 HealthTimeout=10s
 HealthRetries=3
@@ -60,4 +56,4 @@ Restart=always
 RestartSec=10
 
 [Install]
-WantedBy=default.target
+WantedBy=multi-user.target

dankclaw/observability/quadlet/blackbox-exporter.container

@@ -0,0 +1,56 @@
+# Quadlet unit: blackbox exporter (rootless, host network, boot-persistent).
+#
+# Install path on dankclaw:
+#   ~/.config/containers/systemd/blackbox-exporter.container
+#
+# WHY Network=host. The probes it runs are (a) the host's own LAN IP
+# (192.168.0.143:8081 SearXNG, :3005 OpenWebUI — host-process/portmap binds a
+# bridge container cannot reach; netavark DNATs published ports only, see
+# Container Host Reachability in the vault) and (b) ComfyUI's published port
+# via 127.0.0.1 (rootless --network=host shares the real host netns, so the
+# 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
+# reach neither class reliably.
+#
+# WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial
+# its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a
+# bridge container dialing the podman gateway 169.254.1.2 lands on the host
+# LOOPBACK — that is the scrape path, and it needs this exporter to be
+# LISTENING ON LOOPBACK (proved: 200 from the prometheus container to
+# http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway
+# address itself is NOT bindable from the host netns — it exists only
+# inside each rootless netns; binding it dies "bind: cannot assign
+# requested address" (live 2026-10-21). The LAN bind stays for diagnostics.
+# No public DNS name, no guard chain: internal net only, per the exposure
+# policy. The listen flag is --web.listen-address and it is REPEATABLE, one
+# flag per bind; plain --listen-address does not exist and the exporter
+# exits in 100 ms with "unknown long flag" (live 2026-10-21).
+
+[Unit]
+Description=Blackbox exporter (HTTP probe prober)
+After=network-online.target
+
+[Container]
+Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
+ContainerName=blackbox-exporter
+Network=host
+# The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live
+# 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries
+# FLAGS ONLY — repeating the binary makes it a positional arg and kingpin
+# exits "unexpected /bin/blackbox_exporter".
+Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115
+Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+# Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`)
+# is passed as one quoted token and quadlet then folds the following
+# --health-interval/--health-retries (and the image name) into the command
+# string — unit died twice that way, live 2026-10-21. wget only; this
+# busybox variant has no curl, and host-network wget answers on loopback.
+HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy
+HealthInterval=30s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/cadvisor.container

@@ -0,0 +1,56 @@
+# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKCLAW.
+#
+# Install path on dankclaw: ~/.config/containers/systemd/cadvisor.container
+# Unit name: cadvisor.service (container name: cadvisor)
+#
+# WHY THIS EXISTS. The app stack is ~34 rootless Podman containers and until
+# now nothing reported per-container CPU/memory — Grafana could only see the
+# host (node_exporter) and the LLM engines. cadvisor reads the host cgroup
+# tree and exports container_cpu_* / container_memory_* per container.
+#
+# THE LOAD-BEARING FLAG IS --cgroupns=host. Podman starts every container in a
+# PRIVATE cgroup namespace, so an exporter container sees only its OWN cgroup
+# subtree: cadvisor exported a single root series until this flag was added,
+# then it walked the whole tree (996 per-container series measured 2026-10-20).
+# The quadlet unit name is carried in the cgroup path — app.slice/<name>.service/
+# libpod-payload-<hex> — so Prometheus relabels that path into a `container`
+# label (shared/prometheus/prometheus.yml, cadvisor-dankclaw job) and no
+# libpod/podman socket is needed to name containers.
+#
+# BINDING. cadvisor has no per-interface listen flag (unlike node_exporter's
+# --web.listen-address), so it binds 0.0.0.0 INSIDE its own network namespace.
+# The bridge PublishPort is what keeps the unauthenticated endpoint off the
+# host's other interfaces: it publishes exactly loopback (host-local probes and
+# the deployer's health gate) and this host's LAN address (Prometheus, a bridge
+# container, reaches a host-published port at its LAN address — the same netavark
+# rule node-exporter relies on, only here the port really is published).
+#
+# It is rootless on purpose: dank owns the app containers' cgroups, so a
+# rootless cadvisor reads every one of them; root would buy nothing here.
+
+[Unit]
+Description=cAdvisor container metrics (Dankclaw)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=cadvisor
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
+PodmanArgs=--pid=host --cgroupns=host
+PublishPort=127.0.0.1:8080:8080
+PublishPort=192.168.0.143:8080:8080
+Volume=/sys:/sys:ro
+Volume=/:/rootfs:ro
+Exec=--housekeeping_interval=30s
+HealthCmd=wget --spider -q http://localhost:8080/healthz
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=40s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/grafana.container

@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service
 
 [Container]
 ContainerName=grafana
-Image=docker.io/grafana/grafana:latest
+Image=docker.io/grafana/grafana:latest@sha256:dbbc38ea21103cefb50396340cd88afec779c7e070b0a7bf5a75b4ce46651d3f
 Network=dankclaw_default
 User=0:0
 PublishPort=127.0.0.1:3002:3000
@@ -47,9 +47,24 @@ Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
 # client_id query parameter — /application/o/<slug>/authorize/ does not exist.
 Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/
 Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/
-# ?api=1 = introspection-style userinfo: returns custom claims (groups) too.
+# Standard OIDC userinfo. Grafana v13 fetches <api_url>/emails for the address
-Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/?api=1
+# ONLY when the token carries no `email` claim (authentik has no /emails path and
-Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=false
+# 404s it, failing the login) — so the claim must actually be there: the fix is
+# the grafana-email-scope mapping (see scripts/authentik-grafana-sso-fix.sh),
+# not a Grafana flag. Do NOT append ?api=1 to this URL: Grafana then GETs
+# <...>/userinfo/?api=1/emails, a 403.
+Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/
+# authentik's scopes are openid/email/profile. `user:email` (a GitLab-ism) is
+# not a known scope here; requesting it makes the authorize view crash.
+# Quoted: quadlet word-splits Environment=, so an unquoted spaced list
+# truncates SCOPES to its first token.
+Environment="GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile"
+Environment=GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
+Environment=GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=sub
+Environment=GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=preferred_username
+# First SSO login auto-provisions the user. Safe: unauthenticated traffic never
+# reaches this callback — the Anubis -> Authentik guard fronts every request.
+Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
 Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true
 Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true
 Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1

dankclaw/observability/quadlet/node-exporter.container

@@ -33,7 +33,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100

dankclaw/observability/quadlet/prometheus.container

@@ -43,8 +43,16 @@ Wants=network-online.target
 
 [Container]
 ContainerName=prometheus
-Image=docker.io/prom/prometheus:latest
+Image=docker.io/prom/prometheus:latest@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
 Network=dankclaw_default
+# Same-host scrape targets are dialed at host.containers.internal (the podman
+# gateway): a bridge container cannot dial its own host's LAN IP (netavark
+# hairpin gap, measured 2026-10-21), and traffic to the gateway lands on host
+# loopback / published ports (cadvisor/gpu jobs and the blackbox scrape all
+# ride it). The name resolves natively — NO HostAddr key: quadlet 5.7.0 calls
+# it unsupported and the WHOLE user quadlet batch then fails to generate,
+# silently orphaning every other unit (live 2026-10-21: prometheus.service
+# vanished while its container kept running as an orphan).
 User=0:0
 Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
 PublishPort=127.0.0.1:9090:9090

dankclaw/open-webui/quadlet/open-webui.container

@@ -27,7 +27,7 @@ After=network.target
 [Container]
 ContainerName=open-webui
 Network=open-webui.network
-Image=ghcr.io/open-webui/open-webui:main
+Image=ghcr.io/open-webui/open-webui:main@sha256:4860ba005ae8f19f08d1e12b0868855b7b339ea0dad9927f196820fc0de04575
 PublishPort=3005:8080
 Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
 Environment=ANONYMIZED_TELEMETRY=False

dankclaw/pihole/quadlet/pihole.container

@@ -24,7 +24,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=pihole
-Image=docker.io/pihole/pihole:latest
+Image=docker.io/pihole/pihole:latest@sha256:d34134716fc32038b052a8674c7c6879e0ff21f0e6d8af345166fffa2c27c113
 Network=slirp4netns
 PublishPort=0.0.0.0:53:53/udp
 PublishPort=0.0.0.0:53:53/tcp

dankclaw/searxng/quadlet/gluetun.container

@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 [Container]
 ContainerName=gluetun
 Network=searxng.network
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}

dankclaw/searxng/quadlet/searxng-valkey.container

@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 ContainerName=searxng-valkey
 Network=searxng.network
 NetworkAlias=valkey
-Image=docker.io/valkey/valkey:9-alpine
+Image=docker.io/valkey/valkey:9-alpine@sha256:48332870af354a799964c0012ae1194a0bf2bf894eb508f945810596dc2d8d11
 Volume=/opt/searxng/valkey:/data
 
 [Service]

dankclaw/searxng/quadlet/searxng.container

@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
 [Container]
 ContainerName=searxng
 Network=gluetun.container
-Image=ghcr.io/searxng/searxng:latest
+Image=ghcr.io/searxng/searxng:latest@sha256:5ed0323b5f5f8701264ef65e239a2bd83cb57098f2b971fa30bf79c06eeae271
 Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
 Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
 Volume=/opt/searxng/config:/etc/searxng

dankclaw/seerr/quadlet/seerr.container

@@ -29,7 +29,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=seerr
-Image=ghcr.io/seerr-team/seerr:v3.4.1
+Image=ghcr.io/seerr-team/seerr:v3.4.1@sha256:f4768de5f616248d723e05891f3345a1402123775d03bf0890dbfedc0831bda1
 Network=dankclaw_default
 PublishPort=127.0.0.1:5055:5055
 Volume=/home/dank/Containers/seerr/data:/app/config

dankserver/calibre-server/quadlet/calibre-server.container

@@ -43,7 +43,7 @@ After=network.target
 
 [Container]
 ContainerName=calibre-server
-Image=lscr.io/linuxserver/calibre:9.15.0
+Image=lscr.io/linuxserver/calibre:9.15.0@sha256:59e4b0e036e62f228f8bcfae1ade42f47047719885fe454ddbad38e4597b987b
 Network=pasta
 EnvironmentFile=/home/dank/Containers/calibre-server/env
 Environment=PUID=1000

dankserver/calibre-web/quadlet/calibre-web.container

@@ -16,7 +16,7 @@ After=network.target
 
 [Container]
 ContainerName=calibre-web
-Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402
+Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402@sha256:802729966faed5e32d671cd4f92ba6196e9e754a6c0f740ce1ee9acd86b10e55
 Network=pasta
 Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
 Environment=LANG=en_US.UTF-8

dankserver/copyparty/quadlet/copyparty.container

@@ -21,7 +21,13 @@ After=network.target
 
 [Container]
 ContainerName=copyparty
-Image=ghcr.io/9001/copyparty-ac:latest
+# Digest-pinned: upstream stopped publishing version tags after 1.19.15, so
+# the pin rides :latest@digest (the tag in the reference is cosmetic; the
+# digest is the pull). Pinned 2026-10-21 at the build whose label is
+# 1.20.26 - closes GHSA-3c88-p299-7hvr (1.20.25) and GHSA-hg99-jqfp-rf76
+# (1.20.26); the previously running build was the rolling :latest at
+# 1.20.24 (issue #73 on command-center).
+Image=ghcr.io/9001/copyparty-ac:latest@sha256:7f77459160a4bf34fdca79f2965dff4f0c1d7069a3d9cdfb3632acb041dd2f6a
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/copyparty/env
 Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank

dankserver/jellyfin/quadlet/jellyfin.container

@@ -16,7 +16,7 @@ After=network.target
 
 [Container]
 ContainerName=jellyfin
-Image=ghcr.io/jellyfin/jellyfin:latest
+Image=ghcr.io/jellyfin/jellyfin:latest@sha256:c3efa69a22f2691fc88d6bb035e87a8560e395a6047f062fb381c61c3185e8f5
 Network=dankserver_appnet
 Environment=DEBIAN_FRONTEND=noninteractive
 Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true

dankserver/observability/quadlet/cadvisor.container

@@ -0,0 +1,39 @@
+# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKSERVER.
+#
+# Install path on dankserver: ~/.config/containers/systemd/cadvisor.container
+# Unit name: cadvisor.service (container name: cadvisor)
+#
+# The Dankserver twin of dankclaw/observability/quadlet/cadvisor.container.
+# Same rootless shape, same --cgroupns=host requirement (see that file for the
+# full story), only the LAN address differs: this host publishes on
+# 192.168.0.235:8080 so Prometheus on Dankclaw can scrape it over the real
+# network (cross-host, so the host.containers.internal trick does not apply).
+# It reports the media/library stack (romm, plex, jellyfin, copyparty, calibre,
+# tautulli) by their quadlet unit names via the cgroup path.
+
+[Unit]
+Description=cAdvisor container metrics (Dankserver)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=cadvisor
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
+PodmanArgs=--pid=host --cgroupns=host
+PublishPort=127.0.0.1:8080:8080
+PublishPort=192.168.0.235:8080:8080
+Volume=/sys:/sys:ro
+Volume=/:/rootfs:ro
+Exec=--housekeeping_interval=30s
+HealthCmd=wget --spider -q http://localhost:8080/healthz
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=40s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankserver/observability/quadlet/node-exporter.container

@@ -37,7 +37,7 @@ Wants=network-online.target
 
 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100

dankserver/plex/quadlet/plex.container

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
+Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329@sha256:8f655bad7dcbc5398a78f70a5af12600b4ed3424f901566ed1eaa6710e9e2334
 ContainerName=plex
 Network=dankserver_appnet:alias=plex
 Environment=PUID=0

dankserver/romm/quadlet/romm-mariadb.container

@@ -14,7 +14,7 @@ After=network.target
 
 [Container]
 ContainerName=romm-mariadb
-Image=docker.io/library/mariadb:11
+Image=docker.io/library/mariadb:11@sha256:f16d7a344413e519dff518f27bdb221388f84b0d274c97c4fb5f2cb0dc1b34fa
 Network=dankserver_appnet:alias=mariadb
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=MARIADB_DATABASE=romm

dankserver/romm/quadlet/romm-redis.container

@@ -12,7 +12,7 @@ After=network.target
 
 [Container]
 ContainerName=romm-redis
-Image=docker.io/library/redis:7-alpine
+Image=docker.io/library/redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
 Network=dankserver_appnet:alias=redis
 Volume=/home/dank/Containers/romm/redis:/data
 HealthCmd=redis-cli ping

dankserver/romm/quadlet/romm.container

@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target
 
 [Container]
 ContainerName=romm
-Image=ghcr.io/rommapp/romm:latest
+Image=ghcr.io/rommapp/romm:latest@sha256:03295f95a6290e752641f9553ac4198b0e5b71b879d57f7b2371b98f608feb54
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=DB_HOST=mariadb

dankserver/tautulli/quadlet/tautulli.container

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245
+Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245@sha256:c66d3afa8f7a701a48dd1b1f57b22eca0483d895d670fe11ecff783f6a9bbd73
 ContainerName=tautulli
 Network=dankserver_appnet:alias=tautulli
 Environment=PUID=0

frankbook-pro/gluetun/quadlet/gluetun.container

@@ -19,7 +19,7 @@ Description=Gluetun VPN tunnel (FrankBook-Pro podman machine)
 
 [Container]
 ContainerName=gluetun
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}

frankbook-pro/lidarr/quadlet/lidarr.container

@@ -15,7 +15,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=lidarr
-Image=lscr.io/linuxserver/lidarr:latest
+Image=lscr.io/linuxserver/lidarr:latest@sha256:d8549ff9ea29b4015c32257360652c1000c31ea471a2b3f0835ae77292faf35b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/prowlarr/quadlet/prowlarr.container

@@ -24,7 +24,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=prowlarr
-Image=lscr.io/linuxserver/prowlarr:latest
+Image=lscr.io/linuxserver/prowlarr:latest@sha256:b3b00047ed20f6ce2b16780145cb6a0acb51c9e28ea6aa6f5dcd33c7034f1797
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/qbittorrent/quadlet/qbittorrent.container

@@ -29,7 +29,7 @@ Network=gluetun.container
 # Pinned to the last 4.x line: Readarr's retired qB client requires the
 # qB 4.x login response (HTTP 200, body "Ok."); qB 5.x answers 204 with an
 # empty body, which fails Readarr's auth check. All other clients accept 4.x.
-Image=lscr.io/linuxserver/qbittorrent:4.6.7
+Image=lscr.io/linuxserver/qbittorrent:4.6.7@sha256:55f15d44396315551f87294a176efae733b16e283f38980308e46073950257c6
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/radarr/quadlet/radarr.container

@@ -15,7 +15,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=radarr
-Image=lscr.io/linuxserver/radarr:latest
+Image=lscr.io/linuxserver/radarr:latest@sha256:fc0ae3c50f8128b97e84381903ec8d6845c3a569641cb7117c124376cccb57c1
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/readarr/quadlet/readarr.container

@@ -19,7 +19,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=readarr
-Image=lscr.io/linuxserver/readarr:amd64-1.0.980
+Image=lscr.io/linuxserver/readarr:amd64-1.0.980@sha256:a95e82dbb0eb2bb43de4c150f1dab9dd821306c67ea0b418983a957e731f2b61
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

frankbook-pro/sonarr/quadlet/sonarr.container

@@ -16,7 +16,7 @@ After=gluetun.service
 
 [Container]
 ContainerName=sonarr
-Image=lscr.io/linuxserver/sonarr:latest
+Image=lscr.io/linuxserver/sonarr:latest@sha256:889dcc58c6aea859033985894b519b2e6b1e55d92527ed123f9f52e42ae7c97b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles

scripts/apply-protected-cast.sh

@@ -1,27 +1,45 @@
 #!/usr/bin/env bash
-# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
+# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite a recipe's
-# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
+# PROTECTED-BEGIN/END section from the campaign's OWN calibration grams: run
 # derive-protected-experts.py over the calib dir, splice its cast lines into
 # the marked section of the recipe in place, keep the pre-edit recipe in the
-# campaign trash. Refuses to run twice (marker text says derived).
+# campaign trash. Refuses to run twice (marker text says derived), and refuses
+# when a requested target count was not reached. Run it on a COPY of the
+# template so the committed recipe stays the clean, reviewable baseline.
 set -euo pipefail
-CALIB=${1:-/home/dank/corpora/calib}
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
+CALIB=${1:-/home/dank/campaign/qwen38-w5nl64/calib}
-PY=/home/dank/venvs/radcorpus/bin/python
+RECIPE=${2:-$SELF_DIR/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
-DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
+TARGET_COUNT=${3:-0}
+PY=${PY:-/home/dank/venvs/radcorpus/bin/python}
+DERIVE=${DERIVE:-$SELF_DIR/derive-protected-experts.py}
 TS=$(date +%Y%m%d-%H%M%S)
+# OUT and TRASH_DIR are overridable so the seam can drive the whole script in a
+# scratch directory; on the box they are the campaign's usual places.
+OUT=${OUT:-/tmp/protected-cast-$TS.txt}
+TRASH_DIR=${TRASH_DIR:-/home/dank/.AGENTTRASH}
 
 grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
     echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
 grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
 
-"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
+ARGS=("$DERIVE" "$CALIB")
-N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
+if [ "$TARGET_COUNT" -gt 0 ]; then
-[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
+    ARGS+=(--target-count "$TARGET_COUNT")
-echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
+fi
+"$PY" "${ARGS[@]}" > "$OUT"
+N=$(grep -c 'cast dtype=bf16' "$OUT")
+[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing (see $OUT)" >&2; exit 1; }
+if [ "$TARGET_COUNT" -gt 0 ] && [ "$N" -lt "$TARGET_COUNT" ]; then
+    echo "derive reached $N protected experts, target was $TARGET_COUNT; refusing to convert" >&2
+    echo "       (the bar sweep bottomed out: the recording is too thin or the target too high)" >&2
+    exit 1
+fi
+echo "derived $N protected experts -> $OUT"
 
-cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
+mkdir -p "$TRASH_DIR"
-"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
+cp -f "$RECIPE" "$TRASH_DIR/$TS-$(basename "$RECIPE")"
+"$PY" - "$RECIPE" "$OUT" <<'PYEOF'
 import sys
 recipe, castfile = sys.argv[1], sys.argv[2]
 lines = open(recipe).read().splitlines()
@@ -35,7 +53,7 @@ for l in lines:
     elif 'PROTECTED-END' in l:
         insec = False
         out.append(l)
-        out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
+        out.append('# (derive-protected-experts.py rewrote this section from the campaign grams)')
     elif not insec:
         out.append(l)
 open(recipe, 'w').write('\n'.join(out) + '\n')

scripts/authentik-grafana-sso-fix.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Repair the Grafana OAuth2 relying-party SSO on Dankclaw's authentik.
+#
+# grafana.franksdank.cloud logs in through authentik as an OAuth2 relying party.
+# Two DB-side facts (the provider was created by hand via `ak shell`, never
+# repo-managed) silently broke that login, and BOTH had previously been
+# mis-recorded here as a single "the property-mapping evaluator is broken, so
+# userinfo returns only `sub`" myth. They are neither, and the evaluator is fine:
+#
+#   1. provider.authorization_flow was NULL. The authorize view builds a flow
+#      planner from it and crashes with
+#      `AttributeError: 'NoneType' object has no attribute 'slug'` — an HTTP 500
+#      "Server Error — Powered by authentik", before any claim is touched.
+#      Fix: point it at default-provider-authorization-implicit-consent, the
+#      flow every other provider in this instance already uses.
+#   2. The `grafana-email-scope` ScopeMapping's expression was the bare string
+#      `user.email`. authentik merges a scope mapping's result into the token as
+#      a MAPPING: a returned dict adds its keys as claims; a returned string is
+#      keyed under the mapping's NAME ("grafana-email-scope"), so no `email`
+#      claim is ever emitted. Grafana, finding no email, falls back to
+#      GET <api_url>/emails, authentik 404s that path, and the login dies with
+#      "Login failed: InternalError". Fix: return
+#      {"email": user.email, "email_verified": True} — the shape the stock
+#      OpenID 'email' mapping uses. The dict-valued `grafana-profile-scope`
+#      mapping has always worked, which is the proof the evaluator is fine.
+#
+# This is authentik DB state. It is NOT in any deployable file: a blueprint
+# re-import, or an ak-shell re-creation of the grafana provider, reverts BOTH
+# facts and the symptom returns verbatim. Re-run this after any such rebuild.
+# Every step is idempotent.
+#
+# Grafana's own half — SCOPES="openid email profile", a standard userinfo API_URL
+# with no `?api=1`, and ALLOW_SIGN_UP=true — lives in
+# dankclaw/observability/quadlet/grafana.container and deploys normally.
+#
+# Usage: authentik-grafana-sso-fix.sh          (override host: AUTHENTIK_HOST=…)
+set -euo pipefail
+HOST="${AUTHENTIK_HOST:-dankclaw}"
+
+ssh "$HOST" 'podman exec -i authentik-server ak shell' <<'PYEOF'
+from authentik.flows.models import Flow
+from authentik.providers.oauth2.models import OAuth2Provider
+
+p = OAuth2Provider.objects.filter(name="grafana").first()
+if not p:
+    raise SystemExit("no OAuth2Provider named 'grafana'")
+
+# 1. The authorization flow every other provider uses.
+want = Flow.objects.filter(slug="default-provider-authorization-implicit-consent").first()
+if not want:
+    raise SystemExit("default-provider-authorization-implicit-consent flow is missing")
+if p.authorization_flow_id != want.pk:
+    p.authorization_flow = want
+    p.save()
+    print(f"set grafana.authorization_flow -> {want.slug}")
+else:
+    print(f"grafana.authorization_flow already {want.slug}")
+
+# 2. The email scope mapping must RETURN A DICT, not a bare string.
+m = p.property_mappings.filter(name="grafana-email-scope").first()
+if not m:
+    raise SystemExit("grafana-email-scope mapping is missing")
+fixed = 'return {"email": user.email, "email_verified": True}'
+if m.expression.strip() != fixed:
+    m.expression = fixed
+    m.save()
+    print("repaired grafana-email-scope -> returns {email, email_verified}")
+else:
+    print("grafana-email-scope expression already correct")
+
+# 3. The id_token must carry the scope claims; Grafana reads identity from it.
+if not p.include_claims_in_id_token:
+    p.include_claims_in_id_token = True
+    p.save()
+    print("enabled include_claims_in_id_token")
+else:
+    print("include_claims_in_id_token already on")
+PYEOF
+
+echo "Done. Verify: sign in at https://grafana.franksdank.cloud via Authentik;"
+echo "the round trip must land on / and /api/user must answer 200 with an email."

scripts/build-qwen38-bf16-rad.sh

@@ -0,0 +1,220 @@
+#!/usr/bin/env bash
+# build-qwen38-bf16-rad.sh — RUNS ON DANKSERVER. rad-convert the vendor BF16
+# checkpoint /home/dank/models/Qwen3.8-Flash-Next into
+# /home/dank/models/rad/qwen38-bf16.rad with NO recipe and NO --reuse: a weight
+# no rule matches keeps the checkpoint's encoding, so every tensor stays the
+# bf16 it is in the safetensors and the container is the checkpoint's own
+# numbers behind a .rad header.
+#
+# This file is the campaign's measuring stick, twice over: scripts/
+# run-qwen38-bf16.sh kldref records the KLD reference logits from it (every
+# arm's --kld-ref is scored against that record) and the same container serves
+# with the calibration tap to write the Grams that derive the protected-expert
+# list and feed GPTQ. A reference that is not the vendor checkpoint, byte for
+# byte in bf16, silently moves every gate in the campaign — hence no recipe, no
+# --reuse, and the four refusal gates below.
+#
+# Why a container at all: a checkpoint dir has no mapped/file tiers, and a
+# weight only a host kernel reads (the 47.7 GiB PLE n-gram table) cannot be
+# served from one; rad-convert also bakes the tokenizer. The convert runs on
+# the HOST backend — no --device /dev/kfd, no /dev/dri — so declare resolves
+# kernels on CPU and production serving is untouched for the whole run (the
+# precedent that proved it: scripts/convert-swift15-w4dn8.sh).
+#
+# Gates, every one a refusal with exit 2 and a named reason, BEFORE anything is
+# written, cheapest first — the verifier runs last of them because its Hub gate
+# hashes 343 GiB and must not be paid for by a run that was going to refuse on
+# the output file anyway.
+#   checkpoint  the tree must be on disk; an absent dir names the command that
+#               makes one (scripts/download-qwen38-bf16.sh --fetch) and stops.
+#   free space  the convert writes ~343,323 MiB next to the ~343,323 MiB
+#               checkpoint; QWEN38_MIN_FREE_MIB (default 350000 = that figure
+#               plus 6.5 GiB of slack) is the floor. The volume that cannot
+#               hold both is discovered at minute 2, not hour 5.
+#   output      the campaign converts the reference ONCE. An existing
+#               qwen38-bf16.rad is refused; --force retires it to
+#               ~/.AGENTTRASH/<utc>-qwen38-bf16.rad (trash law: nothing is
+#               deleted, and the move is a rename on the same volume).
+#   verified    scripts/download-qwen38-bf16.sh's verify mode must pass. A
+#               failed verify exits 1, not 2: the tree being the wrong payload
+#               is a result, a missing environment is a refusal.
+#   plan        rad-convert --plan-only -v first — the oracle for "does this
+#               shape boot". A refused plan costs seconds and saves a convert.
+#
+# usage: scripts/build-qwen38-bf16-rad.sh [--dry-run] [--plan-only] [--force]
+#   --dry-run    print the argv it would exec (podman wrapping rad-convert),
+#               one shell-quoted argument a line, and exit 0. No gate runs, no
+#               container starts, nothing is written — the offline test seam.
+#               It does resolve the checkpoint and output paths through the
+#               /models mount, because an argv that carries host paths would
+#               not run either.
+#   --plan-only  run the plan and every gate except the disk and output ones —
+#               those two protect bytes the plan never writes, and this is the
+#               probe to run when the disk gate itself is what refuses
+#   --force      retire an existing output container to the trash dir first
+#
+# knobs: FRANK_QWEN38_MODELS   the host dir mounted at /models (default
+#        /home/dank/models); FRANK_QWEN38_CKPT (default $MODELS/
+#        Qwen3.8-Flash-Next, the same knob the verifier reads);
+#        FRANK_QWEN38_BF16_OUT (default $MODELS/rad/qwen38-bf16.rad);
+#        QWEN38_MIN_FREE_MIB; QWEN38_TRASH (default $HOME/.AGENTTRASH). The
+#        verifier's own census knobs (FRANK_QWEN38_SHARDS / _BYTES / _REPO)
+#        reach it through the environment untouched.
+# exit: 0 converted (or planned), 1 the checkpoint failed verification,
+#       2 refused before anything was written
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# ONE owner of the engine pin: the digest frank-archon serves on. The reference
+# must be built by the SAME engine build that scores it, or a KLD delta is
+# partly a version delta.
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-archon.env"
+IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CKPT="${FRANK_QWEN38_CKPT:-$MODELS/Qwen3.8-Flash-Next}"
+OUT="${FRANK_QWEN38_BF16_OUT:-$MODELS/rad/qwen38-bf16.rad}"
+TRASH="${QWEN38_TRASH:-$HOME/.AGENTTRASH}"
+MIN_FREE_MIB="${QWEN38_MIN_FREE_MIB:-350000}"
+LOG="/home/dank/logs/qwen38_bf16_convert.log"
+
+DRY=0
+PLAN_ONLY=0
+FORCE=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --dry-run)   DRY=1 ;;
+        --plan-only) PLAN_ONLY=1 ;;
+        --force)     FORCE=1 ;;
+        *) echo "usage: ${0##*/} [--dry-run] [--plan-only] [--force]" >&2
+           exit 2 ;;
+    esac
+    shift
+done
+
+# The convert sees the tree only through the /models mount, so both paths must
+# be under it; the container-side spelling is what goes into the argv.
+case "$CKPT" in
+    "$MODELS"/*) CKPT_IN="/models/${CKPT#"$MODELS"/}" ;;
+    *) echo "REFUSED: checkpoint $CKPT is not under the mounted models dir $MODELS — the convert cannot see it" >&2
+       exit 2 ;;
+esac
+case "$OUT" in
+    "$MODELS"/*) OUT_IN="/models/${OUT#"$MODELS"/}" ;;
+    *) echo "REFUSED: output $OUT is not under the mounted models dir $MODELS — rad-convert could not write it" >&2
+       exit 2 ;;
+esac
+
+# Declare with NO recipe line — that is what keeps every tensor bf16 — and at the
+# bound this container will actually be served at. Both flags come off the pinned
+# image's own help: --kv-cache-dtype at convert "only picks the kernels it
+# checks" and the container is identical either way, so naming fp8 is free
+# insurance — if the fp8 KV kernels the fleet serves would not resolve, the
+# convert refuses here instead of the boot failing on a drained window.
+# --max-ctx is the bound declare is run at, whose default is the checkpoint's
+# training context (262,144 here, whose fp8 cache is 258.8 GiB against a 192 GiB
+# machine, so the default is not an option). 32,768 is set by the corpus, not by
+# taste: kld.jsonl's longest record is 38,263 characters — ~9,600 tokens at the
+# pessimistic chars/4, ~7,700 at the chars/4.97 this repo measured on the K3
+# corpus — so the window the rig boots at clears every record 2.6x over and the
+# reference boot cannot die on a sequence that will not fit. The rig's window is
+# one knob (FRANK_QWEN38_MAX_MODEL_LEN in scripts/run-qwen38-bf16.sh, default
+# 32,768) and that script refuses a corpus its window cannot hold, so raising one
+# without the other is caught before the drained window, not inside it. The
+# payload containers the gate scores declare at the training context because
+# production must hold the 200k shape; a declared bound is a declare-time check,
+# not a weight change, so the reference and the payloads carry the same numbers
+# whatever each declared at — which is why one can be measured against the other.
+CONVERT_ARGS=("$CKPT_IN" -o "$OUT_IN" --max-ctx 32768 --kv-cache-dtype fp8)
+PODMAN=(podman run --rm --entrypoint rad-convert -v "$MODELS:/models")
+PLAN_CMD=("${PODMAN[@]}" --name qwen38-bf16-plan "$IMAGE" "${CONVERT_ARGS[@]}" --plan-only -v)
+CONV_CMD=("${PODMAN[@]}" --name qwen38-bf16-convert "$IMAGE" "${CONVERT_ARGS[@]}")
+
+if [ "$DRY" = "1" ]; then
+    echo "dry run: no gate checked, no container started, nothing written"
+    echo "# $CKPT -> $OUT (the convert sees them as $CKPT_IN and $OUT_IN)"
+    echo "# plan argv:"
+    printf '%q\n' "${PLAN_CMD[@]}"
+    echo "# convert argv:"
+    printf '%q\n' "${CONV_CMD[@]}"
+    exit 0
+fi
+
+# ---- gate: the checkpoint is on disk ---------------------------------------
+if [ ! -d "$CKPT" ]; then
+    echo "REFUSED: no checkpoint at $CKPT — fetch and verify it first: scripts/download-qwen38-bf16.sh --fetch" >&2
+    exit 2
+fi
+
+# ---- gate: peak disk --------------------------------------------------------
+# The deepest existing ancestor is the right thing to ask: a fresh
+# /home/dank/models/rad has no free space of its own to report.
+OUT_DIR="$(dirname "$OUT")"
+PROBE="$OUT_DIR"
+while [ ! -d "$PROBE" ] && [ "$PROBE" != "/" ]; do PROBE="$(dirname "$PROBE")"; done
+FREE_MIB="$(df --output=avail -m "$PROBE" | tail -1 | tr -d ' ')"
+if ! [[ "$FREE_MIB" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: cannot read the free space under $PROBE (df said: $FREE_MIB)" >&2
+    exit 2
+fi
+if [ "$PLAN_ONLY" != "1" ] && [ "$FREE_MIB" -lt "$MIN_FREE_MIB" ]; then
+    echo "REFUSED peak-disk: $FREE_MIB MiB free under $PROBE, $MIN_FREE_MIB MiB wanted (QWEN38_MIN_FREE_MIB) — the bf16 container rewrites the checkpoint's own 343,323 MiB of tensors beside it" >&2
+    exit 2
+fi
+
+# ---- gate: one conversion ---------------------------------------------------
+if [ -e "$OUT" ]; then
+    if [ "$PLAN_ONLY" = "1" ]; then
+        echo "note: $OUT already exists; --plan-only writes nothing, so it stays"
+    elif [ "$FORCE" = "1" ]; then
+        mkdir -p "$TRASH"
+        DEST="$TRASH/$(date -u +%Y%m%dT%H%M%SZ)-$(basename "$OUT")"
+        mv "$OUT" "$DEST"
+        echo "TRASHED: retired the existing reference container $OUT -> $DEST"
+    else
+        echo "REFUSED: $OUT already exists — the campaign converts the reference once; re-run with --force to retire it to $TRASH first" >&2
+        exit 2
+    fi
+fi
+
+# ---- gate: the tree really is the checkpoint (the full verify) -------------
+echo "== verifying $CKPT (scripts/download-qwen38-bf16.sh, full gates)"
+rc=0
+bash "$SCRIPT_DIR/download-qwen38-bf16.sh" --dir "$CKPT" || rc=$?
+if [ "$rc" -ne 0 ]; then
+    if [ "$rc" -eq 2 ]; then
+        echo "REFUSED: the checkpoint verifier refused the environment (rc=2); nothing was converted" >&2
+        exit 2
+    fi
+    echo "REFUSED: $CKPT failed verification (rc=$rc) — a half tree converts to a half reference" >&2
+    exit 1
+fi
+
+# ---- plan, then convert ------------------------------------------------------
+mkdir -p "$(dirname "$LOG")"
+# rad-convert opens -o during declare as well, so the output directory exists
+# before the plan; that empty directory is all this creates ahead of it.
+mkdir -p "$OUT_DIR"
+echo "== rad-convert --plan-only -v (declare + plan; writes nothing)"
+"${PLAN_CMD[@]}" 2>&1 | tee -a "$LOG"
+rc=${PIPESTATUS[0]}
+if [ "$rc" -ne 0 ]; then
+    echo "REFUSED plan: rad-convert --plan-only exited rc=$rc — the engine refuses this encoding/shape, no container was written (log: $LOG)" >&2
+    exit 2
+fi
+if [ "$PLAN_ONLY" = "1" ]; then
+    echo "PLAN_OK: $CKPT_IN declares for $OUT_IN; no container written (log: $LOG)"
+    exit 0
+fi
+
+echo "== convert: $CKPT -> $OUT ($(date -Is))"
+"${CONV_CMD[@]}" 2>&1 | tee -a "$LOG"
+rc=${PIPESTATUS[0]}
+if [ "$rc" -ne 0 ]; then
+    echo "rad-convert rc=$rc; log: $LOG" >&2
+    exit "$rc"
+fi
+MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
+echo "BUILD_DONE: $OUT is ${MIB} MiB (log: $LOG)"
+echo "NEXT: scripts/run-qwen38-bf16.sh kldref records the reference logits from it"

scripts/convert-qwen38-w5nl64.sh

@@ -0,0 +1,221 @@
+#!/usr/bin/env bash
+# convert-qwen38-w5nl64.sh — RUNS ON DANKSERVER. The CPU half of the Qwen3.8-Flash-Next five-bit
+# campaign: convert the vendor BF16 checkpoint into the campaign's arms, each one the whole
+# `w5nl64a8h` rung on BOTH expert projections with the protected experts kept bf16.
+#
+# Nothing here touches a GPU: the container is started without /dev/kfd or /dev/dri, so the
+# converter runs on the host backend and the fleet can keep serving through the whole script.
+# The engine pin is the digest that serves frank-archon, so a winning arm crosses no version
+# boundary between measurement and serving.
+#
+# Three arms, because the campaign's question is which search knob pays for its hours
+# (recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe):
+#
+#   cd3     the recipe as written — coordinate descent three sweeps. The control.
+#   cd6     six sweeps: the same search, twice the passes.
+#   ao1     act_order=1 instead of cd: GPTQ sweeps columns in activation-importance order and
+#           stores them permuted. This is a SEPARATE arm, not a fourth sweep — libquant refuses
+#           cd= together with act_order=1 ("cd= sweeps the columns in their stored order, and
+#           act_order=1 stores them permuted; one or the other", libquant/lq_registry.cpp), so
+#           the two knobs cannot be combined into a best-of-both container.
+#
+# The arms are generated from ONE template with `--quant` overrides, not three near-duplicate
+# recipes: --quant rules are inserted BEFORE the recipe file and the first rule whose glob matches
+# a weight decides it (core/format/recipe.cpp: Recipe::match), so an override cleanly replaces the
+# expert line and the other 60 lines cannot drift between arms.
+#
+# Protected experts: apply-protected-cast.sh derives the bf16 set from THIS campaign's grams
+# (--target-count 48, two per layer so each rank's 256-expert half stays even) and splices it into
+# a per-run COPY of the template; the committed recipe keeps its markers and stays reviewable.
+#
+# Gates, in the order they bite — each one is a lesson the swift15 campaign paid for:
+#   inputs        checkpoint (dir + index), recipe, grams, imatrix, tokenizer. A missing grams
+#                 directory converts UNCALIBRATED with one stderr note, which is silently worse
+#                 than failing here.
+#   reference     the 335.3 GiB BF16 reference container must be GONE before the converts start.
+#                 It and the checkpoint together eat the volume; its job (the KLD recording and
+#                 the calibration grams) is finished by then, and both survive on disk without the
+#                 container beside it. The FP8-native anchor is FETCHED after the converts (disk
+#                 sequencing), so nothing here waits on it.
+#   plan          --plan-only runs FIRST, per arm, and prints "about X would be written". The
+#                 number is the bill: free space must cover it plus the 128k-prompt overshoot
+#                 margin, and the finished artifact is checked against it.
+#   outputs       no arm overwrites anything; move the old one to trash.
+#
+# Usage: convert-qwen38-w5nl64.sh [--arm cd3|cd6|ao1]... [--dry-run] [--plan-only]
+#                                 [--target-count N] [--keep-recipe]
+set -u -o pipefail
+
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+IMAGE=${IMAGE:-stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed}
+CKPT=${CKPT:-/home/dank/models/Qwen3.8-Flash-Next}
+CAMPAIGN=${CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}
+RECIPE=${RECIPE:-$SELF/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe}
+CALIB=${CALIB:-$CAMPAIGN/calib}
+IMATRIX=${IMATRIX:-$CAMPAIGN/imatrix.gguf}
+OUTDIR=${OUTDIR:-/home/dank/models/rad}
+BF16_RAD=${BF16_RAD:-$OUTDIR/qwen38-bf16.rad}   # the reference container, not an arm output
+STATE=${STATE:-$CAMPAIGN}                      # the dir the container mounts read-only: the
+                                               # spliced recipe and the imatrix live here
+TARGET_COUNT=${TARGET_COUNT:-48}
+MIN_GRAMS=${MIN_GRAMS:-24576}                 # per-expert DOWN grams: 48 recorded layers x 512
+                                               # experts. The checkpoint has 49 MoE layers, but the
+                                               # MTP layer is never recorded, so its 512 experts are
+                                               # absent from the tap's output by design
+MARGIN_MIB=${MARGIN_MIB:-40960}               # the long-prompt overshoot, off the plan number
+TS=$(date +%Y%m%d-%H%M%S)
+
+ARMS=()
+DRY=0
+PLAN_ONLY=0
+KEEP_RECIPE=0
+while [ "$#" -gt 0 ]; do
+    case "$1" in
+      --arm) ARMS+=("$2"); shift 2 ;;
+      --dry-run) DRY=1; shift ;;
+      --plan-only) PLAN_ONLY=1; shift ;;
+      --target-count) TARGET_COUNT="$2"; shift 2 ;;
+      --keep-recipe) KEEP_RECIPE=1; shift ;;
+      *) echo "unknown argument: $1" >&2; exit 2 ;;
+    esac
+done
+[ "${#ARMS[@]}" -gt 0 ] || ARMS=(cd3 cd6 ao1)
+for a in "${ARMS[@]}"; do
+    case "$a" in cd3|cd6|ao1) ;; *) echo "unknown arm: $a (cd3|cd6|ao1)" >&2; exit 2 ;; esac
+done
+
+# The expert rule's option set, spelled once. --quant replaces the template's line wholesale, so
+# every option has to be repeated here; `calib=$CALIB` is expanded by the recipe reader from the
+# container environment (core/format/recipe.cpp: expand refuses an unset name rather than taking
+# it for empty).
+expert_opts='table=w5nl,group=64,scale=fp8_e4m3,scale2=f32,block2=*x*,scale2_value=0.0001220703125,transform=fwht128,rule=search,calib=$CALIB'
+arm_quant() {
+    case "$1" in
+      # The control is the template itself: no override, so the recipe's own line decides.
+      cd3) return 0 ;;
+      cd6) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},cd=6" ;;
+      # act_order=1 WITHOUT cd: libquant refuses the pair (lq_registry.cpp:105, "cd= sweeps the
+      # columns in their stored order, and act_order=1 stores them permuted; one or the other"), so
+      # this arm is the interleave knob alone, not cd=3 plus it.
+      ao1) echo "blk.*.ffn_*_exps.*.weight=gptq:${expert_opts},act_order=1" ;;
+    esac
+}
+
+# --- gates ------------------------------------------------------------------------------------
+[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT; run scripts/download-qwen38-bf16.sh --fetch" >&2; exit 2; }
+[ -f "$CKPT/model.safetensors.index.json" ] || { echo "checkpoint at $CKPT has no safetensors index — a torn fetch" >&2; exit 2; }
+[ -f "$CKPT/tokenizer.json" ] || { echo "no tokenizer.json in $CKPT; the .rad cannot carry one without it" >&2; exit 2; }
+[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 2; }
+GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
+[ "$GRAMS" -ge "$MIN_GRAMS" ] || {
+    echo "calib dir $CALIB holds $GRAMS gram files (<$MIN_GRAMS); run $0 after run-qwen38-bf16.sh calib" >&2; exit 2; }
+[ -f "$IMATRIX" ] || { echo "no imatrix at $IMATRIX; run scripts/grams-to-imatrix.py over $CALIB" >&2; exit 2; }
+# The container sees the campaign dir at one fixed mount point, so an imatrix kept elsewhere has
+# no path inside it: copy it in (or point IMATRIX at the campaign dir) rather than let the
+# converter fail halfway through a planned arm.
+[ "$(dirname "$IMATRIX")" = "$STATE" ] || {
+    echo "imatrix $IMATRIX is not in the campaign dir $STATE the container mounts; move it there" >&2; exit 2; }
+if [ -e "$BF16_RAD" ]; then
+    echo "the 335.3 GiB BF16 reference container $BF16_RAD is still on disk; its recording and" \
+         "the calibration grams are already taken, so move it to trash — the converts need its" \
+         "space (peak-disk law)" >&2
+    exit 2
+fi
+for a in "${ARMS[@]}"; do
+    [ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
+        echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
+done
+
+# --- the protected plane, spliced into a per-run copy of the template --------------------------
+mkdir -p "$STATE"
+WORK="$STATE/recipe-$TS.recipe"
+cp "$RECIPE" "$WORK"
+if [ "$DRY" = 1 ]; then
+    echo "would run: $SELF/apply-protected-cast.sh $CALIB $WORK $TARGET_COUNT"
+else
+    # Positional, because the tool reads its recipe and calib dir by position: passing them as
+    # environment variables would be silently ignored and it would rewrite the COMMITTED
+    # template in place instead of this run's copy.
+    "$SELF/apply-protected-cast.sh" "$CALIB" "$WORK" "$TARGET_COUNT" \
+        || { echo "the protected-expert derivation refused; the arms would convert unprotected" >&2; exit 1; }
+fi
+
+run_convert() { # run_convert ARM [extra flags...]
+    local arm="$1"; shift
+    local out="$OUTDIR/qwen38-w5nl64-$arm.rad" log="$STATE/convert-$arm-$TS.log"
+    local argv=(--rm --name "qwen38-convert-$arm"
+                -e "CALIB=/data/calib" -e "OMP_NUM_THREADS=$(nproc)"
+                -v "$(dirname "$CKPT"):/models:ro"
+                -v "$CALIB:/data/calib:ro"
+                -v "$STATE:/campaign:ro"
+                -v "$OUTDIR:/out")
+    local conv=(/models/"${CKPT##*/}" --recipe "/campaign/$(basename "$WORK")"
+                --tokenizer "/models/${CKPT##*/}/tokenizer.json"
+                --imatrix /campaign/"${IMATRIX##*/}"
+                # No --max-ctx: the context declare then runs at the checkpoint's training context
+                # (262144), which is what the shipped container did — rad-info --meta on it carries
+                # no override. A container declared at a smaller bound cannot be expected to serve
+                # the 200k window D2 holds, and the convert is not where to discover that.
+                # --kv-cache-dtype changes nothing in the artifact; it picks the kernels the declare
+                # checks, and the fleet serves fp8 KV, so an unresolvable fp8 kernel must be said no
+                # to by the convert, not by a boot on a drained window.
+                --kv-cache-dtype fp8 -o "/out/$(basename "$out")")
+    local q; q=$(arm_quant "$arm")
+    [ -n "$q" ] && conv+=(--quant "$q")
+    if [ "$DRY" = 1 ]; then
+        echo "podman run ${argv[*]} \$IMAGE --entrypoint /opt/radiance/bin/rad-convert ${conv[*]} $*"
+        return 0
+    fi
+    podman run "${argv[@]}" --entrypoint /opt/radiance/bin/rad-convert "$IMAGE" "${conv[@]}" "$@" \
+        2>&1 | tee "$log"
+    return "${PIPESTATUS[0]}"
+}
+
+# --- plan, then bill, then convert -------------------------------------------------------------
+overall=0
+for arm in "${ARMS[@]}"; do
+    echo "== arm $arm =="
+    if [ "$DRY" = 1 ]; then
+        # Both lines, in order: the plan runs before the convert, and the convert is what the
+        # operator is about to pay an afternoon for.
+        run_convert "$arm" --plan-only
+        run_convert "$arm"
+        continue
+    fi
+    rc=0; plan_log=$(run_convert "$arm" --plan-only) || rc=$?
+    [ "$rc" -eq 0 ] || { echo "arm $arm: --plan-only failed (rc=$rc); nothing was written" >&2; overall=1; continue; }
+    echo "$plan_log"
+    # "about 141.2 GiB would be written" is the converter's own bill; parse it rather than
+    # hardcoding a size, so the gate still works after any recipe edit.
+    want=$(printf '%s' "$plan_log" | sed -n 's/.*about \([0-9.]*\) \([KMGT]iB\) would be written.*/\1 \2/p' | tail -1)
+    [ -n "$want" ] || { echo "arm $arm: the plan printed no size; refusing to convert blind" >&2; overall=1; continue; }
+    need_mib=$(awk -v n="${want% *}" -v u="${want#* }" \
+        'BEGIN{ f = (u=="KiB") ? 2^-10 : (u=="MiB") ? 1 : (u=="GiB") ? 1024 : (u=="TiB") ? 1048576 : 0
+               if (f == 0) { exit 1 }; printf "%d", n*f + 0.5 }') || {
+        echo "arm $arm: the plan printed an unknown unit in '$want'" >&2; overall=1; continue; }
+    free_mib=$(df --output=avail -m "$OUTDIR" | tail -1 | tr -d ' ')
+    echo "plan: $want ($need_mib MiB); free ${free_mib} MiB on $OUTDIR (margin $MARGIN_MIB MiB)"
+    if [ "$free_mib" -lt $((need_mib + MARGIN_MIB)) ]; then
+        echo "arm $arm: needs ${need_mib} MiB + ${MARGIN_MIB} margin, ${free_mib} free; free space and re-run" >&2
+        overall=1; continue
+    fi
+    if [ "$PLAN_ONLY" = 1 ]; then echo "planned, not converted (--plan-only)"; continue; fi
+    rc=0; run_convert "$arm" || rc=$?
+    [ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
+    out="$OUTDIR/qwen38-w5nl64-$arm.rad"
+    got_mib=$(( $(stat -c %s "$out") / 1048576 ))
+    # The receipt: the artifact is what the plan predicted, within the metadata the plan does not
+    # count. An arm that came out materially smaller lost a plane somewhere.
+    lo=$(( need_mib * 98 / 100 )); hi=$(( need_mib * 103 / 100 ))
+    if [ "$got_mib" -lt "$lo" ] || [ "$got_mib" -gt "$hi" ]; then
+        echo "arm $arm: $out is ${got_mib} MiB, the plan said ${need_mib} MiB (allowed ${lo}-${hi})" >&2
+        overall=1; continue
+    fi
+    echo "CONVERT_DONE $arm ${got_mib} MiB -> $out"
+done
+
+if [ "$KEEP_RECIPE" = 0 ] && [ "$DRY" = 0 ]; then
+    mkdir -p "$CAMPAIGN/.AGENTTRASH"
+    mv "$WORK" "$CAMPAIGN/.AGENTTRASH/$(basename "$WORK")" 2>/dev/null || true
+fi
+exit "$overall"

scripts/deploy-plan.py

@@ -34,12 +34,16 @@ SERVICE_ORDER = (
     "hermes",
     "hermes-dashboard",
     "hermes-webui",
-    "prometheus",
     "grafana",
     "loki",
     "alloy",
     "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
     "alertmanager",
+    # The prober must answer before prometheus re-runs the services job.
+    "blackbox-exporter",
+    "prometheus",
     "authentik-postgres",
     "authentik-server",
     "authentik-worker",
@@ -92,7 +96,10 @@ DANKCLAW_SERVICES = {
     "loki",
     "alloy",
     "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
     "alertmanager",
+    "blackbox-exporter",
     "authentik-postgres",
     "authentik-server",
     "authentik-worker",
@@ -124,6 +131,8 @@ DANKSERVER_SERVICES = {
     "readarr",
     "frank-archon",
     "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
 }
 
 # FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
@@ -271,6 +280,8 @@ def services_for_path(path: str, target: str) -> set[str]:
             # Provisioned datasources and dashboards: git is the source of
             # truth, so a change here must restart the service that reads them.
             _add(services, "grafana")
+        elif group == "blackbox":
+            _add(services, "blackbox-exporter")
         elif group == "homepage":
             # Homepage is a Dankclaw-owned deployment. Shared homepage files
             # are intentionally not copied during a Dankserver-only run.
@@ -374,7 +385,7 @@ def services_for_path(path: str, target: str) -> set[str]:
             # consumer (prometheus before grafana, loki before alloy).
             if len(parts) >= 4 and parts[2] == "quadlet":
                 name = parts[3].removesuffix(".container")
-                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
+                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
                     _add(services, name)
             return services
         if len(parts) >= 2 and parts[1] == "gitea":
@@ -417,8 +428,8 @@ def services_for_path(path: str, target: str) -> set[str]:
             # node-dankserver job had nothing to scrape.
             if len(parts) >= 4 and parts[2] == "quadlet":
                 name = parts[3].removesuffix(".container")
-                if name == "node-exporter":
+                if name in {"node-exporter", "cadvisor"}:
-                    _add(services, "node-exporter")
+                    _add(services, name)
             return services
         # frank-archon's quadlet-era directory left the tree on 2026-09-26
         # (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
@@ -449,6 +460,12 @@ def services_for_path(path: str, target: str) -> set[str]:
             # change to it redeploying alertmanager is the only path that
             # reaches it on the host.
             _add(services, "alertmanager")
+        if name in {"gpu-exporter.py", "gpu-exporter.service"}:
+            # The host-process GPU exporter is installed directly by the
+            # gpu-exporter deploy on BOTH hosts, so its script and unit select
+            # that service (target_services then keeps it only on the host being
+            # planned). Editing the collector restarts the running exporter.
+            _add(services, "gpu-exporter")
         if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
             # llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
             # files here; each one selects frank-zealot only. The ik_llama.cpp

scripts/deploy-remote.sh

@@ -86,7 +86,7 @@ service_is_allowed() {
         dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
         dankclaw:comfyui|dankclaw:hermes|\
         dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:alertmanager|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|dankclaw:blackbox-exporter|\
         dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
         dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -97,7 +97,7 @@ service_is_allowed() {
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
         dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
-        dankserver:frank-archon|dankserver:node-exporter|\
+        dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
         frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
         frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
         frankbook-pro:readarr)
@@ -252,16 +252,24 @@ install_signal_relay() {
     # The relay binds the host's LAN IP (the one address a bridge container
     # can reach as host.containers.internal), so keep the unauthenticated
     # endpoint off the rest of the LAN: admit loopback, the host itself
-    # (its own health probe) and the podman bridge subnet only. Idempotent:
+    # (its own health probe) and the podman bridge subnet only.
-    # -C before -I.
+    # ORDER IS THE GATE: `iptables -I` inserts at position 1, so the old
+    # shape (-C/-I x3 ACCEPT, then -C/-I the DROP) landed the blanket DROP
+    # ABOVE the ACCEPTs and every probe timed out beside a LISTENing socket
+    # (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT; the whole
+    # dankclaw drain aborted at the alertmanager gate twice before the cause
+    # was read off `nft list ruleset`). Rebuild the tail instead of inserting
+    # it: delete any blanket DROP wherever it sits, keep the three ACCEPTs
+    # at the head (-C before -I, unchanged), then APPEND the DROP so it is
+    # last and the ACCEPTs decide first. Converges on poisoned hosts.
+    run_privileged iptables -D INPUT -p tcp --dport 18093 -j DROP 2>/dev/null || true
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT
     run_privileged iptables -C INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT 2>/dev/null \
         || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT
-    run_privileged iptables -C INPUT -p tcp --dport 18093 -j DROP 2>/dev/null \
+    run_privileged iptables -A INPUT -p tcp --dport 18093 -j DROP
-        || run_privileged iptables -I INPUT -p tcp --dport 18093 -j DROP
     # The interpreter needs a moment to bind after restart; a one-shot curl
     # raced the socket on 2026-09-17 and failed a healthy deploy.
     local i
@@ -275,14 +283,81 @@ install_signal_relay() {
     return 1
 }
 
+# Host-process GPU metrics exporter (amdgpu sysfs + nvidia-smi). Not a quadlet:
+# it runs python3 from the checkout on the host so it can read the DRM/cgroup
+# sysfs and shell out to nvidia-smi. Same two-address bind posture as
+# node-exporter/cadvisor (loopback + this host's LAN), so it is never a wildcard.
+deploy_gpu_exporter() {
+    local unit_dir="$HOME/.config/systemd/user" lan dropin i
+    case "$TARGET" in
+        dankclaw) lan=192.168.0.143 ;;
+        dankserver) lan=192.168.0.235 ;;
+        *) echo "gpu-exporter: unknown target '$TARGET'" >&2; return 2 ;;
+    esac
+    mkdir -p "$unit_dir"
+    cp scripts/gpu-exporter.service "$unit_dir/gpu-exporter.service"
+    dropin="$unit_dir/gpu-exporter.service.d"
+    mkdir -p "$dropin"
+    printf '[Service]\nEnvironment=GPU_EXPORTER_BINDS=127.0.0.1,%s\n' "$lan" \
+        > "$dropin/bind.conf"
+    systemctl --user daemon-reload
+    systemctl --user enable gpu-exporter.service
+    systemctl --user restart gpu-exporter.service
+    # The interpreter needs a moment to bind after restart (the same race that
+    # once failed a healthy signal-relay deploy).
+    for i in $(seq 1 15); do
+        curl -fsS -m 5 http://127.0.0.1:9401/metrics >/dev/null && return 0
+        sleep 1
+    done
+    echo "gpu-exporter is not answering on 127.0.0.1:9401/metrics" >&2
+    return 1
+}
+
 copy_loki_config() {
     run_privileged mkdir -p /opt/loki/config
     run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
 }
 
 copy_alloy_config() {
-    run_privileged mkdir -p /opt/alloy/config
+    # The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
-    run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
+    # carries the host's name, so Loki streams say WHICH machine's journal a
+    # line came from without a second config file per host.
+    # *_CONF_ROOT are test seams: the bats cases execute these copiers for
+    # real against a sandbox with a stubbed run_privileged.
+    local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
+    tmp="$(mktemp)"
+    sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
+    if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
+        echo "alloy config: host-label substitution failed" >&2
+        rm -f "$tmp"
+        return 1
+    fi
+    # Syntax gate: alloy's config language has no schema — a misplaced
+    # attribute (e.g. `on_error` outside the json stage) parses only at RUN
+    # time, and the unit then crash-loops with a healthy-looking deploy.
+    # Validate against the very image that will run it, before installing.
+    # Skipped only in the bats sandbox (no image there).
+    if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
+        if ! podman run --rm -v "$tmp:/c.alloy:ro" \
+                --entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
+                validate /c.alloy; then
+            echo "alloy config: validate FAILED against the pinned image" >&2
+            rm -f "$tmp"
+            return 1
+        fi
+    fi
+    run_privileged mkdir -p "$root/config"
+    run_privileged cp "$tmp" "$root/config/config.alloy"
+    rm -f "$tmp"
+}
+
+copy_blackbox_config() {
+    # Single-file mount target: the unit binds this exact path into
+    # /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
+    # BLACKBOX_CONF_ROOT is the same test seam.
+    local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
+    run_privileged mkdir -p "$root"
+    run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
 }
 
 copy_grafana_provisioning() {
@@ -291,14 +366,25 @@ copy_grafana_provisioning() {
     # datasource would be invisible to review, lost with the data volume, and
     # unknown to anyone reading the repository. The unit already mounts
     # /opt/grafana/provisioning read-only, so this only has to populate it.
-    run_privileged mkdir -p /opt/grafana/provisioning/datasources \
+    # GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
-        /opt/grafana/provisioning/dashboards
+    # copy runs unprivileged (the bats case executes this function for real).
+    local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
+    run_privileged mkdir -p "$root/provisioning/datasources" \
+        "$root/provisioning/dashboards"
     run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
-        /opt/grafana/provisioning/datasources/prometheus.yml
+        "$root/provisioning/datasources/prometheus.yml"
     run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
-        /opt/grafana/provisioning/dashboards/provider.yml
+        "$root/provisioning/dashboards/provider.yml"
-    run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
+    # EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
-        /opt/grafana/provisioning/dashboards/llm-fleet.json
+    # turns each file in the directory into a dashboard, so a copier that names
+    # files one by one silently drops new boards (measured 2026-10-21: the
+    # containers board existed in the repo since its commit and was NEVER
+    # provisioned — the panel-census browser check could not find it while the
+    # hand-placed older boards happened to be on disk). Same rule as the quadlet
+    # tier: the directory is the contract, the copy is generated state.
+    for dash in shared/grafana/provisioning/dashboards/*.json; do
+        run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
+    done
 }
 
 render_caddyfile() {
@@ -487,13 +573,15 @@ seed_grafana_env() {
         || { echo "grafana env missing GRAFANA_OAUTH_CLIENT_SECRET; set it in dankclaw/.env" >&2; return 1; }
 }
 
-# The five observability containers left compose on 2026-09-11 (they were
+# The observability containers left compose on 2026-09-11 (they were started by
-# started by hand with podman-compose and never came back after a reboot).
+# hand with podman-compose and never came back after a reboot). Five are ordinary
-# They are ordinary rootless Quadlet units now: same images, same mounts, same
+# rootless Quadlet units: same images, same mounts, same arguments, same
-# arguments, same container names, so nothing that addresses them by name —
+# container names, so nothing that addresses them by name — Caddy, Prometheus'
-# Caddy, Prometheus' own scrape config, Alloy's loki.write endpoint — changed.
+# own scrape config — changed. Only the published ports did: the three HTTP UIs
-# Only the published ports did: the three HTTP UIs bind 127.0.0.1, because the
+# bind 127.0.0.1, because the compose services published on 0.0.0.0 and none of
-# compose services published on 0.0.0.0 and none of them authenticates.
+# them authenticates. Alloy is the exception: a ROOT system service on
+# Network=host, because reading the host journal needs privileges a rootless
+# container cannot get. It is handled by deploy_alloy, not the loop below.
 deploy_observability() {
     local service="$1"
     # Observability units live under <target>/observability/quadlet: Dankclaw
@@ -512,16 +600,17 @@ deploy_observability() {
             copy_loki_config
             build_compose_image loki
             ;;
-        alloy)
-            copy_alloy_config
-            build_compose_image alloy
-            ;;
         grafana)
             copy_grafana_provisioning
             seed_grafana_env || return 1
             ;;
         node-exporter)
             ;;
+        cadvisor)
+            ;;
+        blackbox-exporter)
+            copy_blackbox_config
+            ;;
         *)
             echo "unknown observability service: $service" >&2
             return 2
@@ -534,11 +623,75 @@ deploy_observability() {
         alertmanager) wait_for_health alertmanager http://127.0.0.1:9093/-/healthy 60 ;;
         grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
         loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
-        alloy) wait_for_health alloy http://127.0.0.1:12345/-/healthy 60 ;;
         node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
+        cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
+        # Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address
+        # is not bindable from the host netns — see the unit header).
+        blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
     esac
 }
 
+# Mirror an image the deployer built rootless into the ROOT image store. A root
+# quadlet reads root storage; a root build (build_compose_image is rootless) is
+# not possible on this host — the build container's resolver fails DNS — so the
+# only way to get a root-readable copy is save-and-load. Compares image ids so a
+# rebuilt alloy actually re-lands in root storage instead of leaving a stale tag.
+mirror_image_to_root() {
+    local ref="$1" rid
+    rid="$(podman image inspect "$ref" --format '{{.Id}}' 2>/dev/null)" || {
+        echo "ERROR: rootless image $ref is not built; cannot mirror to root" >&2
+        return 1
+    }
+    if [[ "$(run_privileged podman image inspect "$ref" --format '{{.Id}}' 2>/dev/null)" == "$rid" ]]; then
+        return 0
+    fi
+    local tar; tar="$(mktemp /var/tmp/alloy-mirror.XXXXXX.tar)"
+    podman save -o "$tar" "$ref"
+    run_privileged podman load -i "$tar"
+    rm -f "$tar"
+}
+
+# Install a quadlet into the SYSTEM generator dir (root manager) instead of the
+# user dir. Boot persistence is the unit's own [Install] section (multi-user
+# target); the v1 system generator creates the .wants on daemon-reload, so —
+# like the user units — no `systemctl enable` (generated units cannot be enabled
+# here).
+install_system_quadlet() {
+    local source_dir="$1" unit="$2"
+    run_privileged mkdir -p /etc/containers/systemd
+    run_privileged cp "$source_dir/$unit.container" /etc/containers/systemd/
+    run_privileged systemctl daemon-reload
+}
+
+# Alloy as a ROOT system service on Network=host. See the unit file: a rootless
+# container cannot read the 0640 journal files, and it cannot reach Loki's
+# 127.0.0.1:3100 publish from a bridge netns. Root + host network fixes both;
+# the price is that this one container runs as root and on the host network.
+deploy_alloy() {
+    copy_alloy_config
+    build_compose_image alloy                      # rootless build (resolver works)
+    mirror_image_to_root localhost/franksdank/alloy:latest
+
+    # Retire the rootless unit BEFORE the system one starts: both managers
+    # generate alloy.service, and two containers named "alloy" both binding
+    # 127.0.0.1:12345 cannot coexist. Move the user quadlet out of the generator
+    # dir (it regenerates from there) and stop the user unit.
+    if [[ -f "$quadlet_unit_dir/alloy.container" ]] ||
+       systemctl --user is-active --quiet alloy.service 2>/dev/null; then
+        systemctl --user stop alloy.service 2>/dev/null || true
+        if [[ -f "$quadlet_unit_dir/alloy.container" ]]; then
+            local trash="$HOME/.AGENTTRASH/$(date +%Y%m%d)-alloy-root-cutover"
+            mkdir -p "$trash"
+            mv -- "$quadlet_unit_dir/alloy.container" "$trash/alloy.container"
+            echo "retired rootless alloy.container -> $trash"
+        fi
+        systemctl --user daemon-reload
+    fi
+
+    install_system_quadlet dankclaw/observability/quadlet alloy
+    run_privileged systemctl restart alloy.service
+    wait_for_health alloy http://127.0.0.1:12345/-/healthy 60
+}
 build_compose_image() {
     local service="$1" image cfhash oldhash containerfile
     # Every file the image COPIES IN, beside the Containerfile. The skip below
@@ -1310,9 +1463,10 @@ deploy_anubis() {
 # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
 # qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
 # + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
-# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
+# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
-# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
+# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
-# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
+# run a 200,000 window with its ~45k fit gap; the pool still carries two
+# 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
 # ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
 # relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
 # file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
@@ -1941,7 +2095,9 @@ deploy_service() {
         copyparty) deploy_copyparty ;;
         plex) deploy_plex ;;
         tautulli) deploy_tautulli ;;
-        prometheus|grafana|loki|alloy|node-exporter|alertmanager) deploy_observability "$service" ;;
+        alloy) deploy_alloy ;;
+        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
+        gpu-exporter) deploy_gpu_exporter ;;
         comfyui) deploy_comfyui ;;
         *) deploy_compose_service "$service" ;;
     esac

scripts/deploy.sh

@@ -133,13 +133,15 @@ fi
 valid_service() {
     local service="$1"
     case "$TARGET:$service" in
+        # dankclaw:searxng-network is client-only: a network quadlet with no
+        # remote dispatch handler, so the remote gate does not (must not) list it.
         dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
         dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
         dankclaw:gitea-server|dankclaw:homepage|\
         dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
         dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
-        dankclaw:alertmanager|\
+        dankclaw:alertmanager|dankclaw:blackbox-exporter|\
-        dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
+        dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
         dankclaw:open-notebook|\
         dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
         dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
-        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
+        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
-        dankserver:frank-archon|\
+        dankserver:prowlarr|dankserver:frank-archon|\
         frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
         frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
         frankbook-pro:readarr)

scripts/derive-protected-experts.py

@@ -20,9 +20,16 @@ of the Gram is its down-input energy. Rank halves: 0-255, 256-511
 Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
 
 Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
-       [--min-route-mult 3.0]
+       [--min-route-mult 3.0] [--target-count 0]
+       --target-count N fills the byte bill: the bars pick the base set exactly as
+       the bar-only pass does, then the energy ranking admits experts one at a time
+       (each with its parity partner) until the count lands. The energy bar NEVER
+       relaxes, so asking for 48 bf16 experts never quietly buys 80; the ratio bar
+       walks down only when the ranking cannot produce that many candidates.
+       0 takes the bars as given.
 """
 import argparse
+import collections
 import glob
 import os
 import re
@@ -32,6 +39,7 @@ import sys
 import numpy as np
 
 NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+CAST_RE = re.compile(r"^blk\.(\d+)\.ffn_\*_exps\.(\d+)\.weight")
 
 
 def read_gram(path):
@@ -49,27 +57,8 @@ def read_gram(path):
     return k, rank, int(rows), tr
 
 
-def main():
+def select(data, min_share, min_route_mult):
-    ap = argparse.ArgumentParser()
+    """(cast lines, report lines, count) for one pair of bars."""
-    ap.add_argument("calib_dir")
-    ap.add_argument("--min-share", type=float, default=0.03)
-    ap.add_argument("--min-route-mult", type=float, default=3.0)
-    a = ap.parse_args()
-
-    # layer -> global expert -> (rank, rows, energy)
-    data = {}
-    for path in sorted(glob.glob(os.path.join(
-            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
-        m = NAME_RE.match(os.path.basename(path))
-        if not m:
-            continue
-        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
-        k, _, rows, tr = read_gram(path)
-        data.setdefault(layer, {})[expert] = (rank, rows, tr)
-
-    if not data:
-        raise SystemExit("no per-expert down grams found")
-
     lines, report = [], []
     total = 0
     for layer in sorted(data):
@@ -83,7 +72,7 @@ def main():
         for e, (rank, rows, tr) in experts.items():
             es, rs = tr / tot_e, rows / max(tot_r, 1)
             rec = (e, rank, es, rs)
-            if es >= a.min_share and es >= a.min_route_mult * rs:
+            if es >= min_share and es >= min_route_mult * rs:
                 qual.append(rec)
             elif es > 0:
                 pool.append(rec)
@@ -107,10 +96,148 @@ def main():
                           f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
                           f"ratio {es/max(rs,1e-12):6.1f}")
         total += len(chosen)
+    return lines, report, total
+
+
+def parity_repair(chosen, by_half, report, only=None):
+    """Add the next-best candidate to every half — or to `only` — holding an odd count.
+
+    The four-bit GEMM takes each half's remainder as two equal tables, so an odd count
+    of bf16 experts on a half is not a style problem: the two tables would not match.
+    `by_half` is energy-descending, so the first unchosen candidate is the next most
+    energetic, which is the expert worth keeping. Returns how many were added.
+    """
+    added = 0
+    for key in ([only] if only is not None else sorted(by_half)):
+        layer, half = key
+        n = sum(1 for (l, x) in chosen if l == layer and x // 256 == half)
+        if n % 2 == 0:
+            continue
+        for e, es, rs in by_half.get(key, []):
+            if (layer, e) not in chosen:
+                chosen[(layer, e)] = (es, rs)
+                added += 1
+                report.append(f"blk.{layer}: half {half} odd -> added e{e} "
+                              f"(energy {es*100:.2f}%)")
+                break
+        else:
+            report.append(f"blk.{layer}: half {half} holds {n} and no candidate is "
+                          f"left to make it even")
+    return added
+
+
+def select_to_count(data, min_share, min_route_mult, target):
+    """(lines, report, total, cutoff) for the set that LANDS the count.
+
+    The bars decide the base set; the count is then filled from the energy ranking,
+    one expert at a time, each addition taking its half's next-best candidate with it
+    so every rank's half stays even. Only the RATIO bar walks down to find candidates
+    — it is the law that says an expert is disproportionate and not merely busy, and
+    the caller's byte bill has to be filled — while the ENERGY bar holds, because that
+    bar is what the extra GiB of bf16 is supposed to buy. `cutoff` is the energy share
+    the set bottoms out at: the bar this campaign actually converted with.
+    """
+    report = []
+    shares, ranked = {}, []
+    for layer in sorted(data):
+        experts = data[layer]
+        tot_e = sum(v[2] for v in experts.values())
+        tot_r = sum(v[1] for v in experts.values())
+        if tot_e <= 0 or tot_r <= 0:
+            report.append(f"blk.{layer}: empty recording, skipped")
+            continue
+        for e, (_rank, rows, tr) in experts.items():
+            es, rs = tr / tot_e, rows / max(tot_r, 1)
+            if es <= 0:
+                continue
+            shares[(layer, e)] = (es, rs)
+            ranked.append((es, layer, e, rs))
+    ranked.sort(key=lambda r: (-r[0], r[1], r[2]))
+    by_half = {}
+    for es, layer, e, rs in ranked:
+        by_half.setdefault((layer, e // 256), []).append((e, es, rs))
+
+    chosen = {k: v for k, v in shares.items()
+              if v[0] >= min_share and v[0] >= min_route_mult * v[1]}
+    total = len(chosen)
+    total += parity_repair(chosen, by_half, report)
+
+    mult = min_route_mult
+    while True:
+        pool = [(es, l, e, rs) for es, l, e, rs in ranked
+                if es >= mult * rs and (l, e) not in chosen]
+        if len(pool) + total >= target or mult <= 0.0:
+            break
+        mult -= 0.5
+        report.append(f"ratio bar {mult + 0.5:g} leaves {len(pool)} fill candidates "
+                      f"for {target - total} slots; relaxing toward 0")
+    for es, layer, e, rs in pool:
+        if total >= target:
+            break
+        if (layer, e) in chosen:
+            continue
+        chosen[(layer, e)] = (es, rs)
+        total += 1
+        report.append(f"blk.{layer}: count fill -> added e{e} (energy {es*100:.2f}%)")
+        total += parity_repair(chosen, by_half, report, only=(layer, e // 256))
+    lines, cutoff = [], None
+    for (layer, e) in sorted(chosen):
+        es, rs = chosen[(layer, e)]
+        cutoff = es if cutoff is None else min(cutoff, es)
+        lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight         cast dtype=bf16")
+    return lines, report, total, cutoff
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("calib_dir")
+    ap.add_argument("--min-share", type=float, default=0.03)
+    ap.add_argument("--min-route-mult", type=float, default=3.0)
+    ap.add_argument("--target-count", type=int, default=0)
+    a = ap.parse_args()
+
+    # layer -> global expert -> (rank, rows, energy)
+    data = {}
+    for path in sorted(glob.glob(os.path.join(
+            a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
+        m = NAME_RE.match(os.path.basename(path))
+        if not m:
+            continue
+        rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
+        k, _, rows, tr = read_gram(path)
+        data.setdefault(layer, {})[expert] = (rank, rows, tr)
+
+    if not data:
+        raise SystemExit("no per-expert down grams found")
+
+    if a.target_count > 0:
+        lines, report, total, cutoff = select_to_count(
+            data, a.min_share, a.min_route_mult, a.target_count)
+        bar = f"energy cutoff {cutoff:.3g}" if cutoff is not None else "no candidates"
+    else:
+        lines, report, total = select(data, a.min_share, a.min_route_mult)
+        bar = f"min-share {a.min_share:.3g}"
+
+    # The emitted lines ARE the contract with the four-bit GEMM: every rank's half of
+    # every layer must hold an even number of bf16 experts. Judged on the output and
+    # not on the internal sets, so a slip in either pass is caught here rather than at
+    # engine load time.
+    halves = collections.Counter()
+    for rule in lines:
+        m = CAST_RE.match(rule)
+        if m:
+            halves[(int(m.group(1)), int(m.group(2)) // 256)] += 1
+    odd = {key: n for key, n in halves.items() if n % 2}
+    if odd:
+        print("PARITY LAW BROKEN: " + ", ".join(
+            f"blk.{lay} half {half} holds {n} protected experts"
+            for (lay, half), n in sorted(odd.items())), file=sys.stderr)
+        raise SystemExit(2)
 
     print("\n".join(lines))
     print("\n".join(report), file=sys.stderr)
-    print(f"total protected: {total}", file=sys.stderr)
+    print(f"total protected: {total} (target {a.target_count or 'none'}, {bar}, "
+          f"min-route-mult {a.min_route_mult:g})", file=sys.stderr)
 
 
 if __name__ == "__main__":

scripts/download-qwen38-bf16.sh

@@ -0,0 +1,249 @@
+#!/usr/bin/env bash
+# download-qwen38-bf16.sh — RUNS ON DANKSERVER. Fetch and verify the vendor
+# BF16 checkpoint Qwen/Qwen3.8-Flash-Next: the source the five-bit campaign
+# converts, and the source the KLD reference is recorded from.
+#
+# Why the base and not a finetune: the campaign converts to compare against the
+# publisher's own iq4r container, and that container was built from this
+# checkpoint. Base-in, base-out; anything else confounds the gate. (The
+# predecessor campaign ran ukisai/Swift-1.5-Qwen3.8-Flash-Next-BF16, whose key
+# set is identical to this repo's but whose weights are not — none of its
+# calibration grams, KLD reference or protected list carries over.)
+#
+# Why re-fetch instead of reusing: every BF16 copy of this family was destroyed
+# in the 2026-10-15 storage purge — only the 113.6 GiB served .rad survived.
+# 335.3 GiB of safetensors, 131 shards, ungated (the predecessor campaign
+# verified the key set against this repo, so no gate letter is needed).
+#
+# Verification is four gates deep, because the failure this repo actually lived
+# through is a fetch that reported success: `curl -o FILE URL` exits 0 on an
+# HTTP 401/403 and writes the error body as the artifact — three shards of
+# identical tiny size and a cheerful DONE in 20 seconds for 104 GiB.
+#
+#   1. shard census: the file count must equal the expected shard count, so a
+#      partially fetched fragment tree (the .partNofM trap) cannot pass.
+#   2. per-shard completeness, from the shard's OWN safetensors header: the
+#      header declares where the tensor data ends, so every shard's true size is
+#      known offline. A torn tail is named here, by file, instead of surfacing
+#      as a load error five hours later.
+#   3. index coverage: every shard model.safetensors.index.json references
+#      exists and is named.
+#   4. total bytes against the number the Hub declared, then, unless --no-hub,
+#      every file's sha256 against the Hub's LFS record — including the shards
+#      already on disk, so a previous partial download is never trusted.
+#
+# The token never enters a command line (the 2026-09-16 leak law): it is read
+# into HF_TOKEN in the environment, where a local process listing is the only
+# reader. This repo is ungated, so the fetch simply runs without one when no
+# token file exists. scripts/hfget.sh is curl+jq only — no Python, no venv, no
+# hf_transfer — and it drops the pickle/ONNX weight duplicates by itself.
+#
+# usage: download-qwen38-bf16.sh [--fetch] [--no-hub] [--dir DIR]
+#                                [--shards N] [--bytes B] [--repo ID]
+#   --fetch    download first (resumable: re-run to continue), then verify
+#   --no-hub   skip the Hub-side sha256 sweep; the other gates always run
+# exit: 0 verified, 1 verification failed, 2 environment / usage error
+set -euo pipefail
+
+REPO="${FRANK_QWEN38_HF_REPO:-Qwen/Qwen3.8-Flash-Next}"
+LOCAL_DIR="${FRANK_QWEN38_CKPT:-/home/dank/models/Qwen3.8-Flash-Next}"
+# Hub census of 2026-10-27: 144 files, 131 of them .safetensors, together
+# 360,023,351,514 bytes; the shards alone are 360,000,192,888. This gate counts
+# shards only, so a config file edited on the Hub cannot fail it.
+EXPECTED_SHARDS="${FRANK_QWEN38_SHARDS:-131}"
+EXPECTED_BYTES="${FRANK_QWEN38_BYTES:-360000192888}"
+MIN_FREE_MIB="${FRANK_QWEN38_MIN_FREE_MIB:-350000}"
+
+FETCH=0
+HUB=1
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --fetch) FETCH=1 ;;
+        --no-hub) HUB=0 ;;
+        --dir)    LOCAL_DIR="$2"; shift ;;
+        --shards) EXPECTED_SHARDS="$2"; shift ;;
+        --bytes)  EXPECTED_BYTES="$2"; shift ;;
+        --repo)   REPO="$2"; shift ;;
+        *) echo "usage: ${0##*/} [--fetch] [--no-hub] [--dir DIR] [--shards N] [--bytes B] [--repo ID]" >&2
+           exit 2 ;;
+    esac
+    shift
+done
+
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+FAILS=0
+note() { echo "$@"; FAILS=$((FAILS + 1)); }
+
+if [ "$FETCH" = "1" ]; then
+    # Every fetch gate is a hard stop: skipping the download silently and then
+    # verifying a half tree is how a campaign ships a stale payload.
+    HFGET="$SELF_DIR/hfget.sh"
+    if [ ! -f "$HFGET" ]; then
+        echo "REFUSED: the fetch helper is not here: $HFGET" >&2
+        exit 2
+    fi
+    if ! command -v jq >/dev/null 2>&1; then
+        echo "REFUSED: hfget.sh needs jq on PATH" >&2
+        exit 2
+    fi
+    mkdir -p "$LOCAL_DIR"
+    # The campaign's peak-disk law starts at the fetch: the checkpoint and the
+    # reference container built from it are ~335 GiB each, and a tree that does
+    # not fit is discovered hours into a transfer, when it costs the most.
+    free_mib=$(df -m --output=avail "$LOCAL_DIR" | tail -1 | tr -dc '0-9')
+    if [ "${free_mib:-0}" -lt "$MIN_FREE_MIB" ]; then
+        echo "REFUSED: $free_mib MiB free at $LOCAL_DIR; the fetch wants $MIN_FREE_MIB MiB free" >&2
+        echo "         (retire the peak-disk predecessors first — nothing is downloaded, so" >&2
+        echo "         nothing is left half-written)" >&2
+        exit 2
+    fi
+    echo "fetch: $REPO -> $LOCAL_DIR ($free_mib MiB free)"
+    if [ -z "${HF_TOKEN:-}" ]; then
+        CREDSTORE="$HOME/.config/huggingface/credentials/https://huggingface.co"
+        if [ -f "$CREDSTORE" ]; then
+            HF_TOKEN="$(awk '/^ OauthToken: / {print $2; exit}' "$CREDSTORE")"
+        elif [ -f "$HOME/.cache/huggingface/token" ]; then
+            # The legacy single-line token file the hub library writes.
+            HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.cache/huggingface/token")"
+        elif [ -f "$HOME/.hf_token" ]; then
+            HF_TOKEN="$(tr -d '[:space:]' < "$HOME/.hf_token")"
+        fi
+    fi
+    if [ -n "${HF_TOKEN:-}" ]; then
+        export HF_TOKEN
+    fi
+    bash "$HFGET" "$REPO" "$LOCAL_DIR" main
+    unset HF_TOKEN
+fi
+
+if [ ! -d "$LOCAL_DIR" ]; then
+    echo "REFUSED: $LOCAL_DIR does not exist (run with --fetch, or pass --dir)" >&2
+    exit 1
+fi
+INDEX="$LOCAL_DIR/model.safetensors.index.json"
+if [ ! -f "$INDEX" ]; then
+    echo "REFUSED: $INDEX is missing — the checkpoint tree is not addressable without it" >&2
+    exit 1
+fi
+
+echo "== gate 1: shard census ($LOCAL_DIR)"
+SHARDS=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | wc -l)
+if [ "$SHARDS" != "$EXPECTED_SHARDS" ]; then
+    note "FAIL SHARD_COUNT: $SHARDS safetensors files on disk, expected $EXPECTED_SHARDS"
+else
+    echo "  ok: $SHARDS shards"
+fi
+
+echo "== gate 2: every shard is whole, judged by its own header"
+# A safetensors file is [u64 header length][JSON header][tensor data]. The JSON
+# declares where the data ends, so the file's true size is computable offline:
+# 8 + header + the largest end offset. Truncation and error bodies both fall
+# here, and each is reported with the file that caused it.
+while IFS= read -r f; do
+    state=$(python3 - "$f" <<'PY'
+import json, os, struct, sys
+path = sys.argv[1]
+size = os.path.getsize(path)
+try:
+    with open(path, "rb") as fh:
+        (hlen,) = struct.unpack("<Q", fh.read(8))
+        if hlen == 0 or hlen > size:
+            print("NOT_SAFETENSORS")
+            raise SystemExit
+        header = json.loads(fh.read(hlen))
+except Exception:
+    print("NOT_SAFETENSORS")
+    raise SystemExit
+if not header:
+    print("NOT_SAFETENSORS")
+    raise SystemExit
+end = max((v["data_offsets"][1] for v in header.values()
+           if isinstance(v, dict) and "data_offsets" in v), default=0)
+want = 8 + hlen + end
+print("OK" if want == size else f"TRUNCATED want={want} have={size}")
+PY
+)
+    case "$state" in
+        OK) ;;
+        NOT_SAFETENSORS*) note "FAIL NOT_SAFETENSORS: $(basename "$f") is not a safetensors file (an error body written as an artifact?)" ;;
+        TRUNCATED*) note "FAIL TRUNCATED: $(basename "$f") $state" ;;
+    esac
+done < <(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' | sort)
+if [ "$FAILS" -eq 0 ]; then
+    echo "  ok: every shard's data extent matches its size on disk"
+fi
+
+echo "== gate 3: the index's shards are all present"
+INDEX_SHARDS=$(python3 - "$INDEX" <<'PY'
+import json, sys
+map_ = json.load(open(sys.argv[1]))["weight_map"]
+print("\n".join(sorted(set(map_.values()))))
+PY
+)
+while IFS= read -r name; do
+    [ -n "$name" ] || continue
+    if [ ! -f "$LOCAL_DIR/$name" ]; then
+        note "FAIL MISSING: $name is named by the index and is not on disk"
+    fi
+done <<< "$INDEX_SHARDS"
+if [ "$FAILS" -eq 0 ]; then
+    echo "  ok: $(printf '%s\n' "$INDEX_SHARDS" | grep -c .) referenced shards on disk"
+fi
+
+echo "== gate 4: total bytes vs the Hub's declaration"
+TOTAL=$(find "$LOCAL_DIR" -maxdepth 1 -type f -name '*.safetensors' -printf '%s\n' | awk '{s+=$1} END{print s+0}')
+if [ "$TOTAL" != "$EXPECTED_BYTES" ]; then
+    note "FAIL TOTAL_BYTES: shards total $TOTAL bytes, expected $EXPECTED_BYTES"
+else
+    echo "  ok: $TOTAL bytes"
+fi
+
+if [ "$HUB" = "1" ]; then
+    echo "== gate 5: every file's sha256 vs the Hub's LFS record"
+    # size = file size; lfs.sha256 = sha256 of the file itself (LFS or not).
+    TMPD="$(mktemp -d)"
+    trap 'rm -rf "$TMPD"' EXIT
+    MANIFEST="$TMPD/tree.json"
+    PLAN="$TMPD/plan"
+    curl -sf "https://huggingface.co/api/models/$REPO/tree/main?recursive=true&expand=true" \
+        -o "$MANIFEST" || { echo "FAIL: the Hub API call failed — refusing to report success" >&2; exit 1; }
+    python3 - "$MANIFEST" > "$PLAN" <<'PY'
+import json, sys
+for e in json.load(open(sys.argv[1])):
+    if e.get("type") != "file":
+        continue
+    sha = (e.get("lfs") or {}).get("sha256") or e.get("sha") or ""
+    if sha:
+        print(e["path"], e["size"], sha)
+PY
+    BAD=0
+    while read -r rel size sha; do
+        [ -n "${rel:-}" ] || continue
+        if [ ! -f "$LOCAL_DIR/$rel" ]; then
+            note "FAIL MISSING: $rel is on the Hub and not on disk"
+            continue
+        fi
+        have=$(stat -c %s "$LOCAL_DIR/$rel")
+        if [ "$have" != "$size" ]; then
+            note "FAIL SIZE: $rel is $have bytes, Hub says $size"
+            continue
+        fi
+        got=$(sha256sum "$LOCAL_DIR/$rel" | awk '{print $1}')
+        if [ "$got" != "$sha" ]; then
+            note "FAIL SHA256: $rel got $got want $sha"
+            BAD=$((BAD + 1))
+        fi
+    done < "$PLAN"
+    if [ "$BAD" -eq 0 ] && [ "$FAILS" -eq 0 ]; then
+        echo "  ok: sha256 matches for every file"
+    fi
+    rm -rf "$TMPD"
+    trap - EXIT
+fi
+
+if [ "$FAILS" -gt 0 ]; then
+    echo "VERIFY_FAILED: $FAILS problem(s) in $LOCAL_DIR" >&2
+    exit 1
+fi
+echo "VERIFY_OK: $SHARDS shards, $TOTAL bytes of safetensors in $LOCAL_DIR"
+echo "NEXT: build the BF16 reference container with scripts/build-qwen38-bf16-rad.sh"

scripts/frank-archon.env

@@ -46,31 +46,35 @@ FRANK_ARCHON_TP=2
 # stays EXACT for single-sequence decode and any step of <= 25 rows — see
 # MAX_SEQS x (1 + SPEC_TOKENS) below.
 FRANK_ARCHON_TP_WIRE=wht6
-# Decode slots. A slot costs KV, never decode; the pool is sized to this
+# Decode slots. A slot costs KV and pinned RAM, never decode: the cache target is
-# contract and nothing more. At 1.8.1 the budget resolver caps the cache at
+# exactly MAX_SEQS x MAX_MODEL_LEN and the checkpoint store asks (1 + what the
-# 556,348 fp8 tokens — 2.8 sessions at 200,000 — so a THIRD full-height
+# retention policy keeps) per slot. Ordered 2026-10-20: FOUR slots over a pool of at
-# conversation answers 400 while /health and every shorter chat work fine
+# least 700,000 fp8 tokens, rounding UP if the resolver cannot land it exactly. The
-# (3 x 50k streams run at 77.6 t/s aggregate). Frank's trade: declare the two
+# target is therefore 4 x 200,000 = 800,000 tokens and HOST_POOL_MIB decides whether
-# slots the engine can hold and let the freed ~1.1 GiB go to the expert plane,
+# the cache gets it (expert_vram_floor, core/place/planner.cpp:100-104: the card must
-# instead of advertising a third slot that only exists at shorter prompts.
+# keep expert-plane-minus-host-room). The 2-slot shape was the answer to a
-FRANK_ARCHON_MAX_SEQS=2
+# 365,204-token pool that backed 1.8 full-height sessions; at this pool the third and
+# fourth conversations are backed by the cache instead of over-committing it.
+FRANK_ARCHON_MAX_SEQS=4
 # Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
 # limit, and models.yml contextWindow MUST quote this number (over-length is a
 # refusal, not a truncation). The pool is sized for seqs x maxlen, not one
 # sequence.
 FRANK_ARCHON_MAX_MODEL_LEN=200000
 FRANK_ARCHON_PLACEMENT=expert_tiered
-# Pinned host pool for expert rows that do not fit VRAM. Under the budget
+# Pinned host pool for expert rows that do not fit VRAM, PINNED ONCE PER RANK: the
-# resolver this is ALSO the lever that decides which experts stay on the card:
+# host must pin tp x this number, and the refusal that names it is
-# the card's leftover goes to the expert plane first and the KV cache takes the
+# core/engine_bringup.cpp:250-265. Under the budget resolver this is ALSO the KV
-# rest, so growing the pool moves experts off the card and shrinking it spills
+# lever -- the card keeps expert-plane-minus-this-room, so every MiB added moves one
-# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
+# MiB of experts off the card and into the cache. Derived from the boot ledger (a
-# refused a boot). The ledger's coverage line is the referee, not this file.
+# 23.73 GiB floor at 44,928 room puts plane + 2 units at 67.60 GiB a rank), 49,152
-# Measured, not guessed: 45,568 (PR #30) moved host_pool to 45.09 GiB and left
+# drops the floor to 19.6 GiB and leaves the cache ~6.9 GiB, which is enough to back
-# vram_kv byte-identical at 4.20 GiB / 556,348 tokens on both ranks — extra
+# the whole 4 x 200,000 target; the leftover returns to the experts. Cost is paid
-# host RAM above what the expert plane's footprint needs buys no cache. Back at
+# twice over: pinned RAM (tp x 48 GiB, against the TTM target below) and expert
-# 44,928, the size that boots clean.
+# residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
-FRANK_ARCHON_HOST_POOL_MIB=44928
+# x8 link when the router picks it. The ledger's coverage line is the referee, not
+# this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
+FRANK_ARCHON_HOST_POOL_MIB=49152
 # VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
 # floor is built from it.
 FRANK_ARCHON_GPU_HEADROOM_MIB=96
@@ -93,16 +97,29 @@ FRANK_ARCHON_SPEC_TOKENS=2
 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
 # Step sharing. A prefill chunk that shares a step with decoding sequences holds
 # the device for the chunk's whole time, and every decode row in that step waits
-# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
+# that long for its next token: measured at this shape, decode-only steps 15.5 ms
-# `auto` leaves each decode row behind the chunk a row plus its draft window and
+# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s
-# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
+# while another session prefills (law and census: [[frank-archon]]).
-# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
+# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the
-# chunk took the step and admission never ran. Interleave inserts N decode-only
+# draft window per decoding sequence (3 tokens) and an eighth of the budget per
-# steps after a chunk step taken beside decoders: it protects their cadence and
+# waiting request or waiting prefill, and the source states it will not halve for
-# holds a waiting request out of those steps, so it stays 0 until the post-migration
+# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk
-# decode-vs-prefill buckets say it is wanted.
+# steps. A number is the only value that applies whenever the chunk shares a step,
+# and it must not go under the prefill stager's smallest pass -- 1025 rows here,
+# named on the boot line "prefill staging: ... a step of more than 1024 tokens" --
+# because below it every routed layer reads its pooled experts synchronously, so
+# 512 is a trap and the vendor measures the whole job slower there. Interleave
+# inserts N decode-only steps after a chunk step taken beside decoders: it protects
+# their cadence and holds a waiting request out of those steps. That is the decoder
+# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
+# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
+# price falls on the prefilling prompt (its chunks lose every held step) and on any
+# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
+# ALONE, so it cannot buy this back while decoders run. Meter:
+# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
+# before N rises.
 FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
-FRANK_ARCHON_PREFILL_INTERLEAVE=0
+FRANK_ARCHON_PREFILL_INTERLEAVE=1
 # Prefix cache: pinned host memory only. The disk tier was removed outright
 # (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
 # be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
@@ -118,9 +135,10 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
 # RESETS pages_limit to half of RAM on every boot; below the plan the load dies
 # mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
 # itself before refusing by name, so recovery is a boot, not a visit.
-# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT
+# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
-# per the bats ledger test; at the 44,928 pool that computes to 148,920 MiB.
+# auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
-FRANK_ARCHON_TTM_MIN_MIB=150000
+# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
+FRANK_ARCHON_TTM_MIN_MIB=162000
 FRANK_ARCHON_TTM_TARGET_MIB=204800
 # Escape hatch for experiments. Unset -> no extra argv; the serve script expands
 # this unquoted on purpose, so keep multi-flag values as one string.

scripts/frank-dragoon.env

@@ -49,14 +49,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
 # stays at the image default, which is what was qualified) ------------------
 # Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
 HALOGEN_MODEL_ID=frank-dragoon
-# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
+# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
-# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
+# longer prompt is a 400 naming the limit, never a truncation; stretching
-# the page) — a window equal to the ceiling makes every long request a 400.
+# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
-HALOGEN_CTX=200704
+# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
-# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
+# page): with the window at 200,000 the gap is 62,144, which absorbs the
-# at once. Pool positions buy concurrency; the per-request window never moves
+# chars/4 under-count. A window at or near the ceiling makes every long
-# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
+# request a guaranteed 400.
-# RAM the pool takes is RAM the n-gram table's page cache loses.
+HALOGEN_CTX=262144
+# ONE pool behind both slots. The harness window is 200,000, so 401,408
+# still carries TWO maximum harness-length conversations at once
+# (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
+# ceiling-length requests would queue instead — buying those means raising
+# the pool, and RAM the pool takes is RAM the n-gram table's page cache
+# loses. Pool positions buy concurrency; the per-request window never moves
+# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
 HALOGEN_KV_POOL_POSITIONS=401408
 # Two concurrent conversations: the MTP drafter is not slot-locked on this
 # engine, so the second slot is capacity, not a speculation tax.
@@ -107,10 +114,13 @@ HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
 FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
 # Node paths and the XRT directory are variables so the bats seam can drive
 # these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
-# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
+# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
-# once at its own path, because those libraries find each other at their install
+# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
-# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
+# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
-# system dir, which dangle inside the container.
+# searches that third prefix; without it the lane dies naming it and the
+# supervisor takes the container down — 2026-10-21 boot loop). The basename of
+# this directory IS the multiarch name. The vendor forbids mounting an
+# /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
 FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
 FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
 # sysfs root the fabric-clock preflight reads (same knob the vendor's

scripts/frank-dragoon.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
+Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
 

scripts/gpu-exporter.py

@@ -0,0 +1,212 @@
+#!/usr/bin/env python3
+"""gpu-exporter: unified per-GPU metrics for a mixed AMD/NVIDIA host.
+
+Emits the `gpu_*` Prometheus families the host-system dashboard graphs:
+
+    gpu_utilization_percent  gpu_memory_used_bytes  gpu_memory_total_bytes
+    gpu_temperature_celsius  gpu_power_watts
+
+each labelled {gpu, vendor, model}. The scrape job adds `host`.
+
+Two sources, both host-native — no container, no GPU passthrough:
+
+  * AMD (R9700s, the 8060S iGPU, the Raphael display GPU): the amdgpu sysfs tree.
+    Utilization comes ONLY from gpu_busy_percent — amd-smi reports gfx_busy_inst
+    as N/A on gfx1201 (verified 2026-10-20). VRAM is already bytes, temperature
+    is milli-degrees, power is micro-watts. Crucially power1_average is EMPTY on
+    the R9700s, so those cards legitimately have no gpu_power_watts line; a
+    missing file becomes an ABSENT metric, never a fabricated 0 that a dashboard
+    would plot as a real reading.
+
+  * NVIDIA (the RTX 3090 on Dankclaw): one `nvidia-smi --query-gpu` CSV. On
+    Dankserver there is an nvidia-smi binary but no NVIDIA GPU, so the command
+    fails; that is tolerated and simply yields no NVIDIA series.
+
+Default mode prints one scrape to stdout and exits (the test harness, and any
+`service ... --oneshot` one-off). `--serve` binds the HTTP endpoint for
+Prometheus; it binds EXPLICITLY (loopback + the host LAN address), never a
+wildcard, matching node-exporter's posture on these boxes.
+"""
+import argparse
+import glob
+import os
+import subprocess
+import sys
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+FAMILIES = [
+    ("gpu_utilization_percent", "gauge", "GPU busy time as a percentage of wall time."),
+    ("gpu_memory_used_bytes", "gauge", "Device memory in use, in bytes."),
+    ("gpu_memory_total_bytes", "gauge", "Device memory capacity, in bytes."),
+    ("gpu_temperature_celsius", "gauge", "Package temperature in degrees Celsius."),
+    ("gpu_power_watts", "gauge", "Board power draw in watts."),
+]
+
+DEFAULT_NVIDIA_CMD = (
+    "nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,"
+    "memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits"
+)
+
+
+def _read(path):
+    try:
+        with open(path) as fh:
+            return fh.read().strip()
+    except OSError:
+        return ""
+
+
+def _pci_slot(device_dir):
+    uevent = _read(os.path.join(device_dir, "uevent"))
+    for line in uevent.splitlines():
+        if line.startswith("PCI_SLOT_NAME="):
+            return line.split("=", 1)[1]
+    return ""
+
+
+def _pci_id(device_dir):
+    uevent = _read(os.path.join(device_dir, "uevent"))
+    for line in uevent.splitlines():
+        if line.startswith("PCI_ID="):
+            return "amd:" + line.split("=", 1)[1].lower()
+    return "amd"
+
+
+def collect_amd(sys_dir):
+    for card in sorted(glob.glob(os.path.join(sys_dir, "card[0-9]*"))):
+        device = os.path.join(card, "device")
+        busy = _read(os.path.join(device, "gpu_busy_percent"))
+        if busy == "":
+            continue  # not an amdgpu compute node (e.g. a render node w/o metrics)
+        labels = {"gpu": _pci_slot(device) or os.path.basename(card),
+                  "vendor": "amd", "model": _pci_id(device)}
+        yield "gpu_utilization_percent", labels, busy
+        for name, leaf in (("gpu_memory_used_bytes", "mem_info_vram_used"),
+                           ("gpu_memory_total_bytes", "mem_info_vram_total")):
+            v = _read(os.path.join(device, leaf))
+            if v != "":
+                yield name, labels, v
+        temp = ""
+        for hwmon in sorted(glob.glob(os.path.join(device, "hwmon", "hwmon*", "temp1_input"))):
+            temp = _read(hwmon)
+            if temp != "":
+                break
+        if temp != "":
+            yield "gpu_temperature_celsius", labels, str(int(temp) / 1000.0)
+        power = _read(os.path.join(device, "power1_average"))
+        if power != "":  # empty on the R9700s -> metric absent, not zero
+            yield "gpu_power_watts", labels, str(int(power) / 1e6)
+
+
+def collect_nvidia(cmd):
+    if not cmd:
+        return
+    try:
+        out = subprocess.run(cmd, shell=True, capture_output=True, text=True,
+                             timeout=10, check=True).stdout
+    except (subprocess.SubprocessError, OSError):
+        return  # no NVIDIA GPU / no driver: tolerate, emit nothing
+    for line in out.splitlines():
+        parts = [p.strip() for p in line.split(",")]
+        if len(parts) < 7:
+            continue
+        idx, name, util, mused, mtotal, temp, power = parts[:7]
+        labels = {"gpu": idx, "vendor": "nvidia", "model": name}
+        yield "gpu_utilization_percent", labels, util
+        yield "gpu_memory_used_bytes", labels, str(int(float(mused)) * 1048576)
+        yield "gpu_memory_total_bytes", labels, str(int(float(mtotal)) * 1048576)
+        if temp not in ("", "[N/A]"):
+            yield "gpu_temperature_celsius", labels, temp
+        if power not in ("", "[N/A]"):
+            yield "gpu_power_watts", labels, power
+
+
+def collect(sys_dir, nvidia_cmd):
+    samples = {}
+    for source in (collect_amd(sys_dir), collect_nvidia(nvidia_cmd)):
+        for name, labels, value in source:
+            try:
+                value = str(float(value))  # normalise; drop trailing junk
+            except ValueError:
+                continue
+            samples.setdefault(name, []).append((labels, value))
+    return samples
+
+
+def _fmt(value):
+    f = float(value)
+    if f == int(f) and abs(f) < 1e15:
+        return str(int(f))
+    return repr(f)
+
+
+def render(samples):
+    out = []
+    for name, kind, help_text in FAMILIES:
+        out.append("# HELP %s %s" % (name, help_text))
+        out.append("# TYPE %s %s" % (name, kind))
+        for labels, value in sorted(samples.get(name, []), key=lambda x: sorted(x[0].items())):
+            lset = ",".join('%s="%s"' % (k, v) for k, v in sorted(labels.items()))
+            out.append("%s{%s} %s" % (name, lset, _fmt(value)))
+    return "\n".join(out) + "\n"
+
+
+class Handler(BaseHTTPRequestHandler):
+    sys_dir = "/sys/class/drm"
+    nvidia_cmd = DEFAULT_NVIDIA_CMD
+
+    def do_GET(self):  # noqa: N802 (http.server API)
+        if self.path.split("?")[0] not in ("/metrics", "/"):
+            self.send_error(404)
+            return
+        body = render(collect(self.sys_dir, self.nvidia_cmd)).encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "text/plain; version=0.0.4")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args):
+        pass
+
+
+def main(argv):
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--amdgpu-sysfs", default="/sys/class/drm")
+    ap.add_argument("--nvidia-cmd", default=DEFAULT_NVIDIA_CMD)
+    ap.add_argument("--serve", action="store_true", help="serve /metrics instead of one-shot")
+    ap.add_argument("--port", type=int, default=9401)
+    ap.add_argument("--bind", action="append", default=None,
+                    help="explicit bind address (repeatable); env GPU_EXPORTER_BINDS "
+                         "comma-list if unset. NEVER a wildcard.")
+    args = ap.parse_args(argv)
+
+    if not args.serve:
+        sys.stdout.write(render(collect(args.amdgpu_sysfs, args.nvidia_cmd)))
+        return 0
+
+    binds = args.bind
+    if not binds:
+        env = os.environ.get("GPU_EXPORTER_BINDS", "")
+        binds = [b.strip() for b in env.split(",") if b.strip()] or ["127.0.0.1"]
+    Handler.sys_dir = args.amdgpu_sysfs
+    Handler.nvidia_cmd = args.nvidia_cmd
+    servers = []
+    for b in binds:
+        if b in ("0.0.0.0", "::", "*"):
+            sys.stderr.write("refusing wildcard bind %s\n" % b)
+            return 2
+        servers.append(ThreadingHTTPServer((b, args.port), Handler))
+    sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
+    # EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
+    # healthy in `ss` yet times out every scrape once its SYN backlog fills —
+    # and the loopback deploy gate cannot see it. One thread per extra bind.
+    for s in servers[1:]:
+        threading.Thread(target=s.serve_forever, daemon=True).start()
+    servers[0].serve_forever()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

scripts/gpu-exporter.service

@@ -0,0 +1,35 @@
+# Systemd USER unit: gpu-exporter (both Dankclaw and Dankserver).
+#
+# Install path: ~/.config/systemd/user/gpu-exporter.service
+# Installed and enabled by deploy_gpu_exporter in scripts/deploy-remote.sh.
+#
+# Per-GPU metrics (utilization / VRAM / temperature / power) for the mixed
+# AMD + NVIDIA fleet, scraped by Prometheus. Like signal-relay it is a plain
+# user unit, not a quadlet: it runs python3 straight from the checkout (a pull
+# updates it with no copy step), a hand-written user unit can be enabled where
+# the podman v1 generator's units cannot, and with Linger=yes for dank it starts
+# at boot. It reads the host's own cgroup/drm sysfs and shells to nvidia-smi, so
+# it must run on the host — a container would need GPU/DRM passthrough for
+# nothing gained.
+#
+# BINDING. The base unit serves loopback only. deploy_gpu_exporter writes a
+# drop-in adding THIS host's LAN address (the same two-address posture
+# node-exporter and cadvisor use), so the endpoint is never a wildcard. It is
+# unauthenticated metrics on the LAN, identical to node-exporter :9100 and
+# cadvisor :8080 — Prometheus reaches same-host via host.containers.internal and
+# the Dankserver instance over the LAN from Dankclaw.
+
+[Unit]
+Description=GPU metrics exporter (amdgpu sysfs + nvidia-smi)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+# Loopback by default; the deploy drop-in appends the host LAN address.
+Environment=GPU_EXPORTER_BINDS=127.0.0.1
+ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

scripts/grams-to-imatrix.py

@@ -0,0 +1,418 @@
+#!/usr/bin/env python3
+"""grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
+llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
+beside the recording. Python 3 stdlib only: numpy must NOT be required, because
+this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
+
+WHY. `rad-convert` takes its per-column activation second moments and its
+expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
+Only the GGUF form carries a per-expert dimension — with the legacy flat form,
+or with none, every expert in a layer is weighted by the layer's aggregate,
+which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
+already measured everything; this tool only re-voices it, so the imatrix and the
+protected-expert census (derive-protected-experts.py) are the SAME recording:
+one measurement, two consumers, neither able to disagree about which experts are
+hot.
+
+INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
+(arch/common/rad_arch.h, calib_write). Header law, same parse as
+scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
+u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
+kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
+numbering (rad_block_moe_fp8.h: ge = expert_base + e):
+
+    gram.r<rank>.blk.L.ffn_down_exps.bin        the layer's pooled gram
+    gram.r<rank>.blk.L.ffn_down_exps.<e>.bin    expert e's own gram
+
+`rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
+rows, so **the diagonal is exactly the per-column activation second moment an
+imatrix wants**, and `rows` is the count the reader hands back from `count()`.
+Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
+put a full matrix, and the quantiser asks for columns.
+
+OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
+layer's experts folded into ONE entry, so expert 9 is slice 9 of
+`blk.L.ffn_down_exps`):
+
+  blk.L.ffn_down_exps    in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
+  blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
+                         gate_up input is the same hidden-state rows as its down
+                         input, so the second moment over them is identical and
+                         so is the routing count. Two tensor infos over one
+                         payload: the reader maps each half independently
+                         (imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
+                         by base name), and copying ~200 MB to state one fact
+                         twice buys no correctness.
+  blk.L.ssm_inz          [k] and [1], from the pooled gram: that row's input is
+                         the same hidden state, so its columns take the pooled
+                         diagonal and its count is the layer's pooled rows.
+
+  Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
+  attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
+  entry's cols do not equal the weight's width and the convert proceeds without
+  it: an entry the reader silently ignores is a lie in the artifact, worse than
+  an absent one. The corollary to check before trusting the two hidden-width
+  entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
+  expert width, so the gram's k must equal the checkpoint's hidden width (its
+  config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
+
+INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
+its count stays its own routing rows — the rule the engine itself applies when it
+picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
+once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
+is worse than round-to-nearest, and those experts carry a fraction of a percent
+of the routing). The count never inherits: popularity is `count`'s job. The count
+of inherited experts is reported on stderr.
+
+RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
+so the ranks describe DISJOINT experts and the fold is a UNION into the one
+[k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
+same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
+calib_rows). The same global expert in two per-expert files is legal only when
+the numbers agree exactly; otherwise the recording is corrupt and the entry would
+be whichever file sorted last, so refuse.
+
+GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
+`general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
+`imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
+relative to the aligned data section. Every tensor offset is padded up to
+`general.alignment` and the data section itself starts aligned. That is the
+classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
+arithmetic point into the middle of the previous tensor's floats, and it reads
+them without complaint.
+
+chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
+row total, not a sum over layers: every layer's tap accumulated over the whole
+corpus, so adding the layers counts each token once per layer and inflates the
+figure by the layer count. chunk_count = that row total // chunk_size, the
+"chunks of chunk_size" line imatrix.cpp prints for itself.
+
+REFUSALS, each exit 2 with a message naming what is missing:
+  NO_GRAMS         the directory holds no RADGRAM2 down-projection gram
+  NOT_RADGRAM2     a gram whose magic is not RADGRAM2
+  SHORT            a gram whose length is not 32 + k*k*4
+  NO_POOLED        a layer with expert grams and no pooled gram to inherit from
+  K_MISMATCH       a layer whose files disagree about k (rank pooled grams, or an
+                   expert gram against its layer's pooled gram)
+  EXPERT_CONFLICT  a global expert recorded twice with different numbers
+  EXISTS           the -o path already holds a file (the trash law: move it)
+  CHUNK_COUNT      the derived chunk count does not fit the u32 metadata field
+
+USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
+       [--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
+"""
+import argparse
+import os
+import re
+import struct
+import sys
+
+MAGIC = b"RADGRAM2"
+HDR = 32                       # 8s I I Q I + pad, rad_arch.h
+KOWN_ROWS_PER_K = 4            # libquant/lq_hess.h kOwnRowsPerK
+ALIGNMENT = 32                 # general.alignment, and the pad unit
+
+POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
+EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+
+GT_U32, GT_STR, GT_ARR = 4, 8, 9
+GGML_F32 = 0
+
+
+def refuse(msg):
+    sys.stderr.write("REFUSED %s\n" % msg)
+    sys.exit(2)
+
+
+def read_gram(path):
+    """(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
+    try:
+        with open(path, "rb") as f:
+            raw = f.read()
+    except OSError as e:
+        refuse("UNREADABLE %s: %s" % (path, e))
+    if len(raw) < HDR:
+        refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
+               % (path, len(raw), HDR))
+    magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
+    if magic != MAGIC:
+        refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
+    if k <= 0:
+        refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
+    need = HDR + k * k * 4
+    if len(raw) != need:
+        refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
+               % (path, len(raw), need, k))
+    # The diagonal only, read by stride out of the row-major matrix.
+    diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
+            for i in range(k)]
+    return k, rank, rows, domain, diag
+
+
+def load_dir(d, verbose):
+    """{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
+
+    Files outside the ffn_down_exps stem are not ours to fold: the tap names a
+    pooled gram after every projection it taps, and those carry other widths."""
+    if not os.path.isdir(d):
+        refuse("NO_GRAMS %s: not a directory" % d)
+    layers, rotated = {}, 0
+    for name in sorted(os.listdir(d)):
+        m = POOLED_RE.match(name) or EXPERT_RE.match(name)
+        if not m:
+            continue
+        path = os.path.join(d, name)
+        if not os.path.isfile(path):
+            continue
+        pooled = bool(POOLED_RE.match(name))
+        layer = int(m.group(2))
+        k, rank, rows, domain, diag = read_gram(path)
+        if domain != 0:
+            rotated += 1
+        L = layers.setdefault(layer, {"pooled": None, "experts": {}})
+        if pooled:
+            P = L["pooled"]
+            if P is None:
+                L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
+            elif P["k"] != k:
+                refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
+                       " disagree about the width"
+                       % (layer, P["paths"][0], P["k"], path, k))
+            else:
+                # calib_sum: each rank summed the same stem over its own rows.
+                P["rows"] += rows
+                P["paths"].append(path)
+                for i in range(k):
+                    P["diag"][i] += diag[i]
+            if verbose:
+                sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
+                                 % (layer, name, k, rows))
+            continue
+        e = int(m.group(3))
+        own = L["experts"].get(e)
+        if own is None:
+            L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
+                               "path": path, "ranks": [rank]}
+            if verbose:
+                sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
+                                 % (layer, rank, e, rows))
+            continue
+        if own["rows"] != rows or own["diag"] != diag:
+            refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
+                   " (rows=%d) record the same global expert with different"
+                   " numbers" % (layer, e, own["path"], own["rows"], path, rows))
+        own["ranks"].append(rank)
+        sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
+                         " and %s; folded once\n"
+                         % (layer, e, own["path"], name))
+    if not layers:
+        refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
+               "[.<expert>].bin files in the directory" % d)
+    return layers, rotated
+
+
+def build_entries(layers, min_rows, verbose):
+    """([ (name, dims, payload|None, key) ], census, layers emitted)."""
+    entries = []
+    census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
+    for layer in sorted(layers):
+        P, E = layers[layer]["pooled"], layers[layer]["experts"]
+        if E and P is None:
+            refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
+                   " expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
+                   " under the row threshold would have nothing to inherit"
+                   % (layer, len(E), layer))
+        if P is None:
+            continue
+        k = P["k"]
+        census["layers"] += 1
+        census["rows"] = max(census["rows"], P["rows"])
+        dn = "blk.%d.ffn_down_exps" % layer
+        # A staging key names ONE payload. A gate_up entry reuses the down
+        # layer's bytes by carrying the same key with no payload of its own;
+        # in_sum2 and counts are different bytes and must never share a key.
+        ksum, kcnt = (layer, "sum2"), (layer, "counts")
+        kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
+        # One entry holds one width: an expert gram recorded at another k would
+        # inherit (or be inherited by) a diagonal of the wrong length, and the
+        # reader would then answer `cols` from whichever file set dims[0].
+        for e in sorted(E):
+            if E[e]["k"] != k:
+                refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
+                       " layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
+                                                          E[e]["k"], k))
+
+        if not E:
+            # No per-expert recording for this layer: emit the dense form
+            # ([k], [1]) rather than 512 identical slices, which would claim a
+            # per-expert measurement that was never taken.
+            entries.append((dn + ".in_sum2", [k],
+                            struct.pack("<%df" % k, *P["diag"]), ksum))
+            entries.append((dn + ".counts", [1],
+                            struct.pack("<f", float(P["rows"])), kcnt))
+            sys.stderr.write("note: layer %d has no per-expert grams; %s is"
+                             " dense over the pooled gram\n" % (layer, dn))
+        else:
+            floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
+            n_expert = max(E) + 1
+            sum2, counts = bytearray(), bytearray()
+            for e in range(n_expert):
+                own = E.get(e)
+                if own is not None and own["rows"] >= floor:
+                    src = own["diag"]
+                    rows = own["rows"]
+                else:
+                    src = P["diag"]
+                    rows = own["rows"] if own is not None else 0
+                    census["inherited"] += 1
+                sum2 += struct.pack("<%df" % k, *src)
+                counts += struct.pack("<f", float(rows))
+                census["experts"] += 1
+                census["routing"] += rows
+            if min(E) != 0:
+                sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
+                                 " those slices carry the pooled gram at count 0\n"
+                                 % (layer, min(E) - 1))
+            entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
+            entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
+            gu = "blk.%d.ffn_gate_up_exps" % layer
+            entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
+            entries.append((gu + ".counts", [1, n_expert], None, kcnt))
+            if verbose:
+                sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
+                                 " own-gram floor=%d\n"
+                                 % (layer, k, n_expert, P["rows"], floor))
+
+        inz = "blk.%d.ssm_inz" % layer
+        entries.append((inz + ".in_sum2", [k],
+                        struct.pack("<%df" % k, *P["diag"]), kpgm))
+        entries.append((inz + ".counts", [1],
+                        struct.pack("<f", float(P["rows"])), kpcnt))
+    return entries, census
+
+
+def enc_str(v):
+    b = v.encode("utf-8")
+    return struct.pack("<Q", len(b)) + b
+
+
+def write_gguf(path, entries, datasets, chunk_size, chunk_count):
+    """Write GGUF v3 and return its size in bytes.
+
+    One payload per key: entries that share a key share the bytes and the
+    offset, so a second tensor info states the same fact without copying it."""
+    kv = bytearray()
+    kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", ALIGNMENT)
+    kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", chunk_size)
+    kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", chunk_count)
+    kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
+        + struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
+    for d in datasets:
+        kv += enc_str(d)
+
+    head = bytearray()
+    head += b"GGUF" + struct.pack("<I", 3)
+    head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
+    head += kv
+
+    payloads = {}
+    for name, _dims, payload, key in entries:
+        if payload is None:
+            continue
+        prev = payloads.get(key)
+        if prev is not None and prev != payload:
+            # Two different payloads on one key would ship whichever was
+            # appended last while BOTH tensor infos pointed at it: the counts
+            # payload landing over the in_sum2 payload is exactly how a
+            # hand-rolled GGUF writer loses half its data without an error.
+            refuse("INTERNAL %s: two different payloads share one staging key"
+                   % name)
+        payloads.setdefault(key, payload)
+    infos = bytearray()
+    offsets, order, size = {}, [], 0
+    for name, dims, payload, key in entries:
+        if key not in payloads:
+            refuse("INTERNAL %s: no payload staged for its key" % name)
+        if key not in offsets:
+            offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
+            size = offsets[key] + len(payloads[key])
+            order.append(key)
+        infos += enc_str(name) + struct.pack("<I", len(dims))
+        for d in dims:
+            infos += struct.pack("<Q", d)
+        infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
+
+    body = bytes(head) + bytes(infos)
+    data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
+    with open(path, "wb") as f:
+        f.write(body)
+        f.write(b"\0" * (data_start - len(body)))
+        # The offsets in the tensor infos are the file's law, not a hint: each
+        # payload must land at its own padded offset. Writing them back to back
+        # instead leaves every reader that trusts the info — imatrix.cpp does —
+        # reading its neighbour's floats: plausible values, no error, wrong
+        # quantisation.
+        here = data_start
+        for key in order:
+            f.write(b"\0" * (data_start + offsets[key] - here))
+            f.write(payloads[key])
+            here = data_start + offsets[key] + len(payloads[key])
+        return f.tell()
+
+
+def main():
+    ap = argparse.ArgumentParser(
+        description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
+    ap.add_argument("calib_dir")
+    ap.add_argument("-o", "--out", required=True)
+    ap.add_argument("--datasets", default="",
+                    help="comma-separated names for imatrix.datasets")
+    ap.add_argument("--chunk-size", type=int, default=4096,
+                    help="rows per chunk for imatrix.chunk_size (default 4096)")
+    ap.add_argument("--min-rows", type=int, default=0,
+                    help="own-gram row threshold (default 4*k, the engine's"
+                         " kOwnRowsPerK law)")
+    ap.add_argument("-v", "--verbose", action="store_true")
+    a = ap.parse_args()
+
+    if os.path.exists(a.out):
+        # The repo's deletion law: an artifact that exists is moved to trash by
+        # a human, never silently overwritten by a re-run of the tool that made
+        # it, so a second convert cannot destroy the imatrix the first one fed.
+        refuse("EXISTS %s: an imatrix is already there; move it to"
+               " .AGENTTRASH/ or name another -o" % a.out)
+    if a.chunk_size <= 0:
+        refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
+    layers, rotated = load_dir(a.calib_dir, a.verbose)
+    entries, census = build_entries(layers, a.min_rows, a.verbose)
+    if not entries:
+        refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
+    if rotated:
+        sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
+                         " diagonal is the rotated activation's second moment,"
+                         " not the model's own\n" % rotated)
+
+    chunk_count = census["rows"] // a.chunk_size
+    if chunk_count > 0xFFFFFFFF:
+        # imatrix.chunk_count is a u32 in the reader's grammar, so there is no
+        # legal way to record a bigger one. Reaching this means either a chunk
+        # size picked for a far larger run or a row total read out of the wrong
+        # header offset — rows is a u64 and a u32 parse is off by 2**32 — and
+        # the reader would trust whatever number landed in the field.
+        refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
+               " imatrix.chunk_count field; raise --chunk-size"
+               % (chunk_count, census["rows"], a.chunk_size))
+    nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
+                        a.chunk_size, chunk_count)
+    sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
+                     " rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
+                     % (len(entries), census["layers"], census["experts"],
+                        census["inherited"], census["rows"], census["routing"],
+                        chunk_count, a.chunk_size, nbytes, a.out))
+
+
+if __name__ == "__main__":
+    main()

scripts/kld-gate-qwen38.sh

@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+# kld-gate-qwen38.sh — RUNS ON DANKSERVER (or anywhere the reports are). The campaign's
+# acceptance gate: three KLD reports against ONE reference recording, one verdict.
+#
+# The reference is the BF16 container built from the vendor checkpoint (run-qwen38-bf16.sh
+# kldref, in the same drained window that records the calibration grams). Every scored
+# container is measured against that one recording, so the KL scale runs from 0 = the
+# checkpoint itself downward in faithfulness. Three containers ride it:
+#
+#   fp8.json    the published FP8-native release   — the achievable end of the interval
+#   iq4r.json   the payload frank-archon serves    — the bar to beat
+#   cand.json   the arm under test                 — the subject
+#
+# The FP8 number is what makes the interval absolute instead of relative: "half the gap"
+# is half of the distance between the 4-bit payload and the near-lossless release, not half
+# of an invented distance to the checkpoint. That is also why the FP8 payload is FETCHED and
+# scored, never converted: it is a second opinion about what this checkpoint tolerates, not a
+# step of the campaign. It lands LAST (disk sequencing) — by then the 335 GiB bf16 container
+# is gone, so its 172.8 GiB fits.
+#
+# The law, in order of severity, on the mean KL of one slice (--slice all | a corpus source):
+#   1. KL(fp8) < KL(iq4r)                 the anchors are ordered. If the near-lossless
+#                                         release does NOT sit closer to the checkpoint than
+#                                         the 4-bit payload, the corpus, the tokenizer or the
+#                                         boot shape is wrong and no arm number means anything.
+#   2. KL(fp8) < KL(cand) < KL(iq4r)      the candidate lies strictly inside the interval —
+#                                         outside it, something is mis-measured, not better
+#   3. KL(cand) <= KL(iq4r) - R x gap     it closes at least R (--recover, default 0.5) of the
+#                                         gap = KL(iq4r) - KL(fp8): at least half the way from
+#                                         what is served toward the release (agreed 2026-10-27)
+#   4. top-1(cand) >= top-1(iq4r)         the agreement the harness keys on does not fall
+#
+# Booting the containers is NOT this script's job: `run-qwen38-bf16.sh score RAD OUT.json`
+# writes the reports and owns the single engine argv (same pin, --tp 2, --kv-cache-dtype fp8,
+# same step size, --deterministic, no speculation) that makes the three numbers comparable.
+# This script is arithmetic, so it re-runs against old reports and is testable without a GPU.
+#
+# Usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]
+set -uo pipefail
+
+[ "$#" -ge 3 ] || { echo "usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]" >&2; exit 2; }
+python3 - "$@" <<'PY'
+import json, sys
+
+a = sys.argv[1:]
+opts, files = [], []
+i = 0
+while i < len(a):
+    if a[i].startswith("--"):
+        opts += [a[i], a[i + 1]]
+        i += 2
+    else:
+        files.append(a[i])
+        i += 1
+if len(files) != 3:
+    sys.exit("usage: kld-gate-qwen38.sh FP8.json IQ4R.json CAND.json [--slice NAME] [--recover F]")
+o = dict(zip(opts[0::2], opts[1::2]))
+slice_ = o.get("--slice", "all")
+recover = float(o.get("--recover", 0.5))
+if not 0.0 < recover <= 1.0:
+    sys.exit(f"--recover must be in (0,1], got {recover}")
+
+
+def die(msg):
+    print(msg, file=sys.stderr)
+    raise SystemExit(2)
+
+
+def read(path):
+    try:
+        d = json.load(open(path))
+    except Exception as e:
+        die(f"cannot read the KLD report {path}: {e}")
+    if slice_ == "all":
+        g = d.get("all")
+    else:
+        g = (d.get("by_source") or {}).get(slice_)
+        if g is None:
+            die(f"{path} has no slice {slice_!r}; it has: "
+                f"{', '.join(sorted(d.get('by_source') or {})) or '(none)'}")
+    if g is None or "kld" not in g or "top1_agreement" not in g:
+        die(f"{path} has no {slice_!r} summary — wrong report version?")
+    return d, g
+
+
+rows = []
+for label, path in zip(("fp8", "iq4r", "cand"), files):
+    d, g = read(path)
+    rows.append((label, path, d.get("model", "?"), g["kld"]["mean"], g["kld"]["p99_9"],
+                 g["top1_agreement"], g["positions"]))
+
+print(f"reference: BF16 container; slice: {slice_}")
+print(f"{'':6} {'model':38} {'KL mean':>10} {'KL p99.9':>10} {'top-1':>8}  positions")
+for label, path, model, kl, p999, top1, n in rows:
+    print(f"{label:6} {str(model)[:38]:38} {kl:10.6f} {p999:10.4f} {100*top1:7.2f}%  {n}")
+r = {label: (kl, top1) for label, _p, _m, kl, _q, top1, _n in rows}
+kl_fp8, kl_iq4r, kl_cand = r["fp8"][0], r["iq4r"][0], r["cand"][0]
+t_iq4r, t_cand = r["iq4r"][1], r["cand"][1]
+gap = kl_iq4r - kl_fp8
+ok = True
+
+
+def law(n, text, passed):
+    global ok
+    ok = ok and passed
+    print(f"  [{'PASS' if passed else 'FAIL'}] {n}. {text}")
+
+
+law(1, f"anchors ordered: fp8 {kl_fp8:.6f} < iq4r {kl_iq4r:.6f} (gap {gap:.6f})",
+    gap > 0)
+law(2, f"candidate inside the interval: {kl_fp8:.6f} < {kl_cand:.6f} < {kl_iq4r:.6f}",
+    kl_fp8 < kl_cand < kl_iq4r)
+need = kl_iq4r - recover * gap
+law(3, f"candidate closes {100*max((kl_iq4r - kl_cand)/gap, 0):.1f}% of the "
+       f"{gap:.6f} gap (needs {100*recover:.0f}%, i.e. KL <= {need:.6f})"
+       if gap > 0 else "gap is not positive; cannot score recovery",
+    gap > 0 and kl_cand <= need)
+law(4, f"candidate top-1 {100*t_cand:.2f}% >= served iq4r top-1 {100*t_iq4r:.2f}%",
+    t_cand >= t_iq4r)
+print("KLD GATE " + ("PASS" if ok else "FAIL"))
+sys.exit(0 if ok else 1)
+PY

scripts/probe-frank-archon-kv.py

@@ -147,10 +147,21 @@ def main():
     deep = corpus(160_000)
     wide = corpus(1_100_000)
 
-    # 3.28 chars/token is the measured ratio of this corpus through the
+    # The corpus's chars/token ratio is MEASURED here, not stored. A stored 3.28
-    # froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
+    # (646,000 chars -> 196,919 tokens, 2026-10-11) had drifted to 3.20 by
-    # deploy); 98.5 % of the window leaves the response room under the ceiling.
+    # 2026-10-20 (646,299 chars -> 201,876 tokens) as the repo's docs grew, and a
-    chars = int(0.985 * maxlen * 3.28)
+    # stale constant sends leg B past --max-model-len, where the engine answers a
+    # perfectly legitimate 400 ("the request (201900 tokens) exceeds the available
+    # context size (200000)") and the leg reads as a capacity failure. One small
+    # request prices the ratio; 97 % of the window leaves the response room.
+    sample = prompt(wide, 4000, "CAL", "cal-marker")
+    cal, _, _ = post({"model": MODEL,
+                      "messages": [{"role": "user", "content": sample}],
+                      "max_tokens": 1, "temperature": 0.0,
+                      "chat_template_kwargs": {"enable_thinking": False}})
+    ratio = len(sample) / cal["usage"]["prompt_tokens"]
+    print(f"[cal] corpus ratio {ratio:.2f} chars/token")
+    chars = int(0.97 * maxlen * ratio)
     # Each marker's FIRST hyphen-separated word is unique, because show()
     # scores the hit on that word — a shared prefix would score every slot HIT
     # on any answer that says the word.
@@ -176,7 +187,7 @@ def main():
                 print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
     print(f"[B] total {time.time() - t0:.1f}s")
     # D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
-    dchars = int(0.25 * maxlen * 3.28)
+    dchars = int(0.25 * maxlen * ratio)
     dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
                for i in range(seqs)]
     print(f"[D] {seqs} concurrent streams x 512 tokens, "

scripts/rechunk-kld-corpus.py

@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+"""Re-chunk a KLD corpus so every document fits the container that scores it.
+
+A KLD reference is recorded by booting one container and prefilling every corpus
+document through it. radiance refuses a document the KV pool cannot hold on its
+own (core/mem/kv.cpp `can_ever_fit`: a paged group holds `n_blocks *
+block_size` tokens), and the number that matters is printed by the boot ledger,
+not by a flag:
+
+    W kv group 'kv_attn' holds 7436 tokens but the context limit is 32768, so
+    a prompt longer than 7436 tokens is refused (400) ... or lower
+    --max-model-len to 7436
+
+The BF16 qwen4exp container at the campaign's shape gets 270.00 MiB of cache --
+the expert plane owns both cards -- so its real window is 7,436 tokens however
+`--max-model-len` is spelled. The corpus the campaign inherited from the swift15
+era is 197 documents of 8,193-11,970 tokens, because make-calib-corpus.py's
+`--kld-min-prompt 8192` sets a floor and no ceiling: every one of them is
+unscoreable there, and three drained windows have proved it.
+
+This script moves the document boundaries and nothing else. Each record's token
+ids -- counted with the same tokenizer the generator used -- are cut into
+windows of at most `--max-doc-tokens`, so a record's windows concatenate back to
+that record's own token sequence, unchanged: no token dropped, rewritten or
+reworded. Each window gets its own `score_from = len - --score-tail`, the same
+relation the generator wrote for the parent, so the reference scores 256
+positions per window instead of per document: 100,864 measured positions over
+the same text against the 50,432 the inherited corpus scored, at a shorter
+context. The calibration corpus is untouched, so the quantization itself does
+not move.
+
+Usage:
+  rechunk-kld-corpus.py --corpus kld.jsonl --out kld.jsonl.new \
+      [--tokenizer-json PATH] [--max-doc-tokens 7168] [--score-tail 256] [--force]
+
+--out must not exist and be non-empty without --force: the same
+one-artifact-one-name law run-qwen38-bf16.sh applies to a score report. The file
+is written through a temporary name and renamed, so an interrupted run leaves no
+half corpus behind. A JSON summary of what was written goes to stdout.
+"""
+
+import argparse
+import json
+import os
+import sys
+import tempfile
+
+
+def parse_args() -> argparse.Namespace:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--corpus", required=True,
+                    help="the kld.jsonl to re-chunk (records: prompt, score_from, source)")
+    ap.add_argument("--out", required=True,
+                    help="file to write; refuse if it exists and is non-empty without --force")
+    ap.add_argument("--tokenizer-json",
+                    default=os.environ.get("FRANK_QWEN38_TOKENIZER",
+                                           "/home/dank/models/Qwen3.8-Flash-Next/tokenizer.json"),
+                    help="the tokenizer the corpus was authored with (the same one the "
+                         "manifest in the corpus output dir names)")
+    ap.add_argument("--max-doc-tokens", type=int, default=7168,
+                    help="the widest document the scoring container can hold; 7,168 is "
+                         "7x1024, the largest whole kilopage under the 7,436 the bf16 "
+                         "container's ledger reports")
+    ap.add_argument("--score-tail", type=int, default=256,
+                    help="positions scored at the end of every document, as the generator "
+                         "recorded it")
+    ap.add_argument("--force", action="store_true",
+                    help="overwrite a non-empty --out (after moving the old file to trash)")
+    return ap.parse_args()
+
+
+def load_tokenizer(path: str):
+    try:
+        from tokenizers import Tokenizer
+    except ImportError as exc:  # the corpus scripts carry their own venv
+        raise SystemExit(
+            f"REFUSED: tokenizers is not importable by {sys.executable} — run the "
+            f"corpus venv's python (on dankserver: /home/dank/venvs/radcorpus/bin/python) "
+            f"or set FRANK_QWEN38_TOKENIZER: {exc}") from None
+    if not os.path.isfile(path):
+        raise SystemExit(f"REFUSED: no tokenizer at {path}")
+    return Tokenizer.from_file(path)
+
+
+def refuse(msg: str) -> None:
+    print(f"REFUSED: {msg}", file=sys.stderr)
+    sys.exit(2)  # 2 is the code argparse uses for unusable arguments
+
+
+def main() -> None:
+    a = parse_args()
+    if a.max_doc_tokens <= 0:
+        refuse(f"--max-doc-tokens must be positive (got {a.max_doc_tokens})")
+    if a.score_tail <= 0:
+        refuse(f"--score-tail must be positive (got {a.score_tail})")
+    if a.max_doc_tokens <= a.score_tail:
+        refuse(f"--max-doc-tokens {a.max_doc_tokens} cannot hold a {a.score_tail}-position "
+               "score tail: every window would be all tail and no context")
+    if not os.path.isfile(a.corpus):
+        refuse(f"no corpus at {a.corpus}")
+    if os.path.realpath(a.out) == os.path.realpath(a.corpus):
+        refuse(f"--out {a.out} is the corpus itself; move it to trash first — the trash law "
+               "means the old segmentation stays recoverable")
+    if os.path.exists(a.out) and os.path.getsize(a.out) > 0 and not a.force:
+        refuse(f"{a.out} exists and is not empty; pass --force after moving it to trash")
+
+    tok = load_tokenizer(a.tokenizer_json)
+
+    docs_in = docs_out = tokens_in = tokens_out = 0
+    longest_child = 0
+    clamped = scored = 0
+    sources: dict[str, int] = {}
+    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(os.path.abspath(a.out)) or ".",
+                              prefix=".rechunk-")
+    try:
+        with os.fdopen(fd, "w", encoding="utf-8") as out:
+            with open(a.corpus, encoding="utf-8") as src:
+                for lineno, line in enumerate(src, 1):
+                    if not line.strip():
+                        continue
+                    try:
+                        rec = json.loads(line)
+                    except json.JSONDecodeError as exc:
+                        refuse(f"{a.corpus}:{lineno} is not JSON: {exc}")
+                    prompt = rec.get("prompt")
+                    if not isinstance(prompt, str):
+                        refuse(f"{a.corpus}:{lineno} has no string 'prompt' field")
+                    ids = tok.encode(prompt, add_special_tokens=False).ids
+                    docs_in += 1
+                    tokens_in += len(ids)
+                    for start in range(0, max(len(ids), 1), a.max_doc_tokens):
+                        window = ids[start:start + a.max_doc_tokens]
+                        if not window:
+                            continue
+                        score_from = max(len(window) - a.score_tail, 0)
+                        if len(window) < a.score_tail:
+                            clamped += 1
+                        scored += len(window) - score_from
+                        child = {"prompt": tok.decode(window),
+                                 "score_from": score_from}
+                        if "source" in rec:
+                            child["source"] = rec["source"]
+                        out.write(json.dumps(child, ensure_ascii=False) + "\n")
+                        docs_out += 1
+                        tokens_out += len(window)
+                        longest_child = max(longest_child, len(window))
+                        if "source" in rec:
+                            for cls in str(rec["source"]).split("+"):
+                                sources[cls] = sources.get(cls, 0) + 1
+        if tokens_out != tokens_in:
+            raise SystemExit(f"REFUSED: the windows hold {tokens_out} tokens and the corpus "
+                             f"has {tokens_in} — re-chunking lost or invented tokens")
+        os.replace(tmp, a.out)
+    except BaseException:
+        if os.path.exists(tmp):
+            os.unlink(tmp)
+        raise
+    os.chmod(a.out, 0o644)
+
+    print(json.dumps({
+        "corpus": a.corpus,
+        "out": a.out,
+        "tokenizer": os.path.basename(a.tokenizer_json),
+        "documents_in": docs_in,
+        "documents_out": docs_out,
+        "tokens_in": tokens_in,
+        "tokens_out": tokens_out,
+        "longest_document": longest_child,
+        "max_doc_tokens": a.max_doc_tokens,
+        "score_tail": a.score_tail,
+        "scored_positions": scored,
+        "clamped_score_from": clamped,
+        "source_documents": sources,
+    }, indent=1))
+
+
+if __name__ == "__main__":
+    main()

scripts/recipes/qwen4exp-qwen38-w5nl64-bf16trunk.recipe

@@ -0,0 +1,80 @@
+# qwen4exp-qwen38-w5nl64-bf16trunk.recipe — every routed expert, BOTH projections, at five bits:
+# u5 codes into libquant's 32-level w5nl table, an E4M3 scale a group of 64 under the fixed 2^-13
+# second level, rotated by the 128-wide Hadamard, chosen by GPTQ against $CALIB and swept three
+# times by coordinate descent. This is `w5nl64a8h` in the engine's served-encoding table — the top of
+# the four-bit class — applied wider than any published radiance container of this checkpoint: the
+# author's own w5 rung (`qwen4exp-w5dn64-i8-hc8m.recipe`) moves only `down`, because that rung was
+# priced as a byte-for-error trade against prefill; this campaign buys the whole rung and pays for
+# it out of the pinned host pool (+15.3 GiB of expert plane, `--host-pool-mib 61440` per rank), not
+# out of the KV pool.
+#
+# Every other line is the served container's recipe (dump it with
+# `rad-info --recipe /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`), minus the rows this
+# campaign leaves at the checkpoint:
+#
+#   * the trunk's block linears, the shared expert and the hyper-connection mixers are NOT listed —
+#     dropping their i8/fp8 lines leaves them bf16, which is what the checkpoint holds. Measured bill
+#     +1.978 GiB of always-resident weight = ~131k tokens of KV off the pool.
+#   * `ssm_inz`/`ssm_out` STAY int8: the delta-net gates are the largest trunk rows (+1.9 GiB, 126k
+#     tokens) and the least sensitive per byte.
+#   * `mtp.draft_head` STAYS 2-bit and the PLE n-gram table STAYS one fixed E4M3 scale: the draft
+#     head's rung is cheap and its acceptance cost is a decode question this campaign does not touch,
+#     and bf16 for the n-gram table would pin +47.69 GiB of host RAM for a lookup.
+#
+# Protected experts: none in this file. `scripts/apply-protected-cast.sh` appends the derived set
+# (48, two per layer so each rank's 256-expert half stays even — the four-bit GEMM takes the rest of
+# a half as two equal tables by parity) from the fresh 12M-token energy/routing census, and they must
+# sit ABOVE the expert row because the first rule that matches a weight decides it.
+#
+#   CALIB=/home/dank/campaign/qwen38-w5nl64/calib \
+#     rad-convert /home/dank/models/Qwen3.8-Flash-Next \
+#       --recipe this.file --imatrix .../imatrix.gguf -o out.rad
+#
+# The arm driver (`scripts/convert-qwen38-w5nl64.sh`) generates the per-arm variants of the expert
+# row from this template — cd=3 (control), cd=6, and `act_order=1` with NO cd= (libquant refuses the
+# pair, lq_registry.cpp:105) — rather than keeping three near-duplicate recipes that can drift apart.
+#
+# The whole campaign, in the order that fits on one disk (every step runs on dankserver):
+#
+#   1. scripts/download-qwen38-bf16.sh --fetch      the vendor BF16 checkpoint, 335.3 GiB
+#   2. scripts/build-qwen38-bf16-rad.sh             the BF16 reference container, another 335.3 GiB
+#   3. one drained window, two boots:
+#        scripts/run-qwen38-bf16.sh kldref          the KLD reference recording (the zero of scale)
+#        scripts/run-qwen38-bf16.sh calib           the 12M-position calibration tap -> grams
+#   4. trash qwen38-bf16.rad                        the peak-disk law: the converts need its room
+#   5. scripts/grams-to-imatrix.py                  grams -> imatrix.gguf in the campaign dir
+#   6. scripts/convert-qwen38-w5nl64.sh             the arms, planned and billed before each one runs
+#   7. fetch the published FP8-native release, then per container:
+#        scripts/run-qwen38-bf16.sh score RAD OUT   fp8 anchor, served iq4r, each arm
+#      then scripts/kld-gate-qwen38.sh fp8.json iq4r.json cand.json
+#   8. a winner is a trio edit plus `scripts/deploy.sh dankserver --service frank-archon`; the
+#      rollback is the commit before it. If the win needs the bigger pinned plane, raise
+#      FRANK_ARCHON_HOST_POOL_MIB and FRANK_ARCHON_TTM_TARGET_MIB in the SAME commit: the preflight
+#      floor is 2xHOST_POOL + 48,824 + prefix cache + MNBT + the store, and a pool raised without
+#      the TTM target is a boot refusal that Restart=always repeats forever, invisible to deploy.sh.
+
+# PROTECTED-BEGIN — scripts/apply-protected-cast.sh splices the derived cast lines in here.
+# They MUST sit above the expert rule: the first rule whose glob matches a weight decides it, so a
+# cast line below the five-bit rule would never win. The set is derived from the campaign's own
+# 12M-token recording (`derive-protected-experts.py --target-count 48` over the fresh grams), and the
+# driver splices it into a per-arm COPY so this template stays the clean, reviewable baseline.
+# PROTECTED-END
+
+# Every routed expert, trunk and MTP layer alike, both projections, at five bits. The experts' scales
+# sit between 2^-15 and 2^-10.6 under w4nl's top level of 22 and half a binade lower under w5nl's 30
+# — inside E4M3's normal range under 2^-13 either way. The MTP layer has no recording and is rounded
+# to nearest, which libquant says on stderr; it is one layer of 49 and only ever changes drafts.
+blk.*.ffn_*_exps.*.weight            gptq table=w5nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
+
+# The delta-net's in/out gates, int8 with a bf16 scale a group of 128 searched for least error — the
+# served recipe's lines, unchanged.
+blk.*.ssm_inz.weight                 rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
+blk.*.ssm_out.weight                 rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
+
+# The MTP head's two mixing rows, E4M3 as served. They only ever change the drafts.
+mtp.fc_hidden.weight                 rtn codes=fp8_e4m3 group=128 scale=f32
+mtp.fc_embedding.weight              rtn codes=fp8_e4m3 group=128 scale=f32
+
+# The n-gram table at one fixed E4M3 scale, and the 2-bit draft head — both the served lines.
+blk.*.ple_ngram.weight               rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
+mtp.draft_head.weight                rtn codes=u2 zero=u8 group=128 scale=f16

scripts/run-qwen38-bf16.sh

@@ -0,0 +1,346 @@
+#!/usr/bin/env bash
+# run-qwen38-bf16.sh — RUNS ON DANKSERVER. Boot a container over the campaign's
+# ONE measuring shape for the phases that need both R9700s. The engine owns the
+# cards, so frank-archon is stopped first and restored on every exit path — the
+# autoresearch pattern: stop, never mask (the unit file in /etc/systemd/system
+# is a symlink into the checkout and mask fails on a relinked unit; the deploy
+# restores it).
+#
+#   kldref            --kld-record the reference log-probabilities: the engine
+#                     runs /home/dank/corpora/out/kld.jsonl (197 prompts, 50,432
+#                     scored positions, score tail 256) through prefill, writes
+#                     /home/dank/campaign/qwen38-w5nl64/kld-ref, and EXITS. The
+#                     model behind it is the BF16 container from
+#                     scripts/build-qwen38-bf16-rad.sh — the vendor checkpoint's
+#                     own numbers, which is what makes it the zero point.
+#   score RAD OUT     --kld-ref score ONE container against that recording and
+#                     EXIT: the FP8-native anchor, the served iq4r, and every
+#                     arm. Deliberately the SAME engine pin, the same TP, the
+#                     same step size, the same fp8 KV, --deterministic and no
+#                     speculation as kldref — KL and top-1 agreement are only
+#                     comparable across boots when everything except the weights
+#                     is identical, so this mode is one argv with --model and
+#                     the two --kld-* flags changed, never a second shape.
+#                     RAD must sit under the models dir (the container reads it
+#                     through the ro /models mount); OUT lands on a writable
+#                     mount because --kld-out F also writes F.rows beside it.
+#   calib             serve on :7770 with the calibration tap open
+#                     (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS=12000000). A
+#                     separate driver — scripts/calib-push.py --port 7770
+#                     --container qwen38-bf16 --corpus
+#                     /home/dank/corpora/out/calib.jsonl — pushes the corpus; at
+#                     12M pooled positions the engine writes the Grams and the
+#                     tap switches itself off (calib-push prints TAP_DONE). Stop
+#                     this script after that; the trap brings frank-archon back.
+#
+# Shape: the vendor's bf16 calibration lines — expert_tiered placement with the
+# weights disk tier, deterministic (the resident set pinned to the loader's plan,
+# so a recorded reference and a scored candidate see the same one), no
+# speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM,
+# and the production chat template rather than the one baked into the container —
+# every scored container is asked the same tokenised questions the reference
+# answered. The window the flag spells is 32,768; the pool this payload's boot
+# actually gets is 7,436 tokens, so kld.jsonl is re-chunked by
+# scripts/rechunk-kld-corpus.py into documents of at most 7,168 tokens — the
+# widest one the reference can hold. The kldref gate below re-checks the file
+# against the declared window, and the re-chunker is what keeps that honest.
+#
+# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
+# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
+# removed: the card is split between the weights and the cache from what the
+# model needs." The resolver derives the split from --host-pool-mib,
+# --gpu-headroom-mib and the activation arena; the rig inherits the first two from
+# frank-archon.env, the file that owns them. On THIS payload the host pool is not
+# the cache's supplier: the boot lands in vram_budget.cpp's `no lender` branch,
+# where the experts keep their plan-fixed 25.29 GiB and the cache gets the residue.
+# Measured twice — 12,288 and 49,152 MiB of host pool both printed the same
+# "kv cache 271.25 MiB ... 7,436"; only --gpu-headroom-mib moved anything
+# (768 -> 96 grew vram_weights 29.41 -> 30.06 GiB). The supplier the source names
+# is the arena: vram_budget.cpp:117, "lower --max-num-batched-tokens (the arena
+# scales with it)" — which is what the step size below now does.
+#
+# The pool, not the window, decides what the recorder accepts: it refuses any
+# document longer than the pool, so the 32,768-token window was decorative while the
+# cache held 7,436 — "E kld: document 196 is 8507 tokens and the KV pool holds
+# 7436, so it can never be scored" on all 197 records, exit 1. The longest record
+# is 10,554 tokens by the engine's own count (chars/3.6 for this file, not the
+# chars/4.97 this repo measured on the K3 corpus).
+#
+# The step size is ONE value for the recording and every scoring pass:
+# FRANK_QWEN38_MNBT, default 1,024. The gate's whole claim is that the reference,
+# the served iq4r and every arm rode one shape, so the ceiling is set by the
+# TIGHTEST payload the gate boots rather than by the bf16 reference: the quantised
+# expert FILE tiers (the served iq4r, the arms) stream only at MNBT<=2048 and
+# refuse the widened form above it. 2,048 was still the wrong default on this
+# payload, because the arena the engine charges for a step comes straight out of
+# the cache — 1.08 GiB at a 2,048-token step, 504.39 MiB at a 1,024-token one, and
+# the experts keep their share either way. At 1,024 the cache is 866 MiB: 24,300
+# tokens, which scores every one of the 197 documents (longest 11,970).
+# Calibration keeps the wide 65,536 — grams are not logits, nothing downstream is
+# compared against a step size, and that is where a big step pays.
+#
+# usage: scripts/run-qwen38-bf16.sh kldref|calib|score RAD OUT [--dry-run] [--image REF]
+#   (RAD and OUT are score's two positionals: RAD is a container under the models
+#    dir, OUT the report path whose directory is created if missing)
+#   --dry-run   print the exact `podman run` argv, one shell-quoted argument a
+#               line, and exit 0: the service is NOT stopped, nothing is
+#               mounted, nothing is written — the offline test seam. It still
+#               validates the argv's own shape (paths through the /models mount,
+#               a step size that is a number), because an argv that could not
+#               boot is not worth printing.
+#   --image REF override the digest pin (a probe boot, or the test seam)
+#
+# knobs: FRANK_QWEN38_MODELS (the dir mounted at /models), FRANK_QWEN38_RAD (the
+#        bf16 container, kldref only), FRANK_QWEN38_CAMPAIGN (holds kld-ref/ and
+#        calib/), FRANK_QWEN38_KLD_CORPUS, FRANK_QWEN38_TPL (the template copy
+#        the container reads through /models), FRANK_QWEN38_LOG_DIR, and
+#        FRANK_QWEN38_MNBT — the step size shared by kldref and score (default
+#        1,024: the file tier's ceiling is 2,048 and production runs there, but on
+#        this payload the arena charged for a wider step is taken from the cache),
+#        and FRANK_QWEN38_MAX_MODEL_LEN — the window every mode boots at (32,768;
+#        see the corpus census above, and raise it together with the corpus).
+# kldref refuses (exit 2) when the longest record of the corpus it is about to
+# read is longer than MAX_MODEL_LEN x 3 characters — chars/3 being the
+# pessimistic token:char ratio, so the script says no before the engine finds out
+# inside a drained window, after the checkpoint work is already done. score
+# refuses when the reference dir is absent, when RAD is not on disk or not under
+# the models mount, or when OUT already exists: the trash law means a second
+# score over the same name is a deliberate act, and two numbers in one file is
+# how a campaign loses its own history.
+# Engine stdout goes to $LOG_DIR/qwen38_bf16_<mode>.log — the boot ledger is the
+# record of what the resolver actually gave the cards; the journal never sees it.
+# exit: the engine's code; 2 = refused before the service was touched
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-archon.env"
+IMAGE="${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
+
+MODE=""
+POS=()
+DRY=0
+while [ $# -gt 0 ]; do
+    case "$1" in
+        kldref|calib|score) MODE="$1" ;;
+        --dry-run)          DRY=1 ;;
+        --image)            IMAGE="$2"; shift ;;
+        *)
+            # score's two positional arguments, and nothing else may be bare.
+            if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
+                POS+=("$1")
+            else
+                echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF]  (unknown argument: $1)" >&2
+                exit 2
+            fi ;;
+    esac
+    shift
+done
+if [ -z "$MODE" ]; then
+    echo "usage: ${0##*/} kldref|calib|score RAD OUT [--dry-run] [--image REF]  (no mode given)" >&2
+    exit 2
+fi
+if [ "$MODE" = score ] && [ "${#POS[@]}" -ne 2 ]; then
+    echo "usage: ${0##*/} score RAD OUT  — RAD is the container to score, OUT the report to write (got ${#POS[@]} argument(s))" >&2
+    exit 2
+fi
+
+MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
+CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
+KLD_CORPUS="${FRANK_QWEN38_KLD_CORPUS:-/home/dank/corpora/out/kld.jsonl}"
+RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
+TPL="${FRANK_QWEN38_TPL:-$MODELS/tpl.jinja}"
+TEMPLATE="$SCRIPT_DIR/qwen38-froggeric-v22.5.jinja"
+LOG_DIR="${FRANK_QWEN38_LOG_DIR:-/home/dank/logs}"
+LOG="$LOG_DIR/qwen38_bf16_${MODE}.log"
+KLD_REF_DIR="$CAMPAIGN/kld-ref"
+CALIB_DIR="$CAMPAIGN/calib"
+C=qwen38-bf16
+SERVICE=frank-archon
+# The card split, taken from the file that owns it instead of a private pair:
+# --host-pool-mib also funds the KV cache (see the header), so hardcoding it
+# here once starved the recorder. The `:?` guards keep an empty env fatal.
+HOST_POOL_MIB="${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
+GPU_HEADROOM_MIB="${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
+
+# ONE step size for both logits modes. The file tier streams only at <=2,048 and
+# the gate scores file-tier payloads (the served iq4r, the arms) against the
+# reference, so the reference is recorded at a step the payloads can meet — never
+# two numbers that a KL between them would have to forgive. 1,024 rather than the
+# 2,048 ceiling: on this payload the arena charged for the step is the cache's only
+# remaining supplier, and 1,024 is the rung that funds every corpus document.
+MNBT="${FRANK_QWEN38_MNBT:-1024}"
+if ! [[ "$MNBT" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: FRANK_QWEN38_MNBT must be a token count (got: $MNBT)" >&2
+    exit 2
+fi
+
+# The window every mode boots at. Over-length is a REFUSAL, not a truncation (the
+# env file says so for production), so this is the number the corpus must fit in.
+MAX_MODEL_LEN="${FRANK_QWEN38_MAX_MODEL_LEN:-32768}"
+if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
+    echo "REFUSED: FRANK_QWEN38_MAX_MODEL_LEN must be a token count (got: $MAX_MODEL_LEN)" >&2
+    exit 2
+fi
+
+# Every container this campaign boots is read through the ro /models mount, so
+# the host path must live under it; the container-side spelling is what goes
+# into the argv. Same rule for the container a score pass opens.
+under_models() {
+    case "$1" in
+        "$MODELS"/*) printf '/models/%s' "${1#"$MODELS"/}" ;;
+        *) return 1 ;;
+    esac
+}
+
+# same_dir A B — the two name one directory, judged only when BOTH exist: a dry
+# run has no fixtures on disk, and two absent paths are not the same place.
+same_dir() {
+    [ -d "$1" ] && [ -d "$2" ] && [ "$(cd "$1" && pwd)" = "$(cd "$2" && pwd)" ]
+}
+
+if [ "$MODE" = score ]; then
+    SCORE_RAD="${POS[0]}"
+    SCORE_OUT="${POS[1]}"
+    OUT_DIR="$(dirname "$SCORE_OUT")"
+    MODEL_IN="$(under_models "$SCORE_RAD")" || {
+        echo "REFUSED: container to score $SCORE_RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
+        exit 2
+    }
+else
+    MODEL_IN="$(under_models "$RAD")" || {
+        echo "REFUSED: reference container $RAD is not under the mounted models dir $MODELS — the engine cannot read it" >&2
+        exit 2
+    }
+fi
+
+# The measuring shape. Identical for kldref and score by construction — this
+# array is the only place the campaign's KL numbers are defined.
+ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
+      --host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
+      --deterministic --num-speculative-tokens 0
+      --kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
+      --ngram-placement ram
+      --override-chat-template /models/tpl.jinja)
+MOUNTS=(-v "$MODELS:/models:ro")
+TAPENV=()
+case "$MODE" in
+  kldref)
+    # Recording is prefill-only over four slots at the campaign's single step size
+    # — 1,024, not production's 2,048 (frank-archon.env: FRANK_ARCHON_MAX_SEQS=4,
+    # FRANK_ARCHON_MAX_BATCHED_TOKENS=2048): the wider step costs the cache the
+    # room the corpus needs. Corpus read-only, record dir rw.
+    ARGS+=(--model "$MODEL_IN"
+           --kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
+           --max-num-seqs 4 --max-num-batched-tokens "$MNBT")
+    MOUNTS+=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
+             -v "$KLD_REF_DIR:/data/kld-ref")
+    ;;
+  score)
+    # The same argv with three words changed: which container, and that this
+    # time the engine reads a recording and writes a report instead of the
+    # reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
+    # able to touch the zero point — and the report gets its own writable bind.
+    if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR"; then
+        echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
+        exit 2
+    fi
+    ARGS+=(--model "$MODEL_IN"
+           --kld-ref /data/kld-ref --kld-out "/data/out/$(basename "$SCORE_OUT")"
+           --max-num-seqs 4 --max-num-batched-tokens "$MNBT")
+    MOUNTS+=(-v "$KLD_REF_DIR:/data/kld-ref:ro"
+             -v "$OUT_DIR:/data/out")
+    ;;
+  calib)
+    # Calibration is exempt from the shared step: the tap counts pooled
+    # positions and nothing downstream is compared against a step size, so the
+    # wide step is free throughput (a disk-tier step costs the same at 1 token
+    # as at 64k).
+    ARGS+=(--model "$MODEL_IN" --port 7770
+           --max-num-seqs 8 --max-num-batched-tokens 65536)
+    MOUNTS+=(-v "$CALIB_DIR:/data/calib")
+    TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
+            -e RADIANCE_CALIB_TOKENS=12000000)
+    ;;
+esac
+
+CMD=(podman run --rm --name "$C"
+     --network host --init
+     --security-opt seccomp=unconfined
+     --ulimit memlock=-1:-1
+     --device /dev/kfd --device /dev/dri
+     "${MOUNTS[@]}" "${TAPENV[@]}"
+     "$IMAGE" "${ARGS[@]}")
+
+# ---- gate: the window must hold the corpus (kldref alone can meet it) --------
+# score reads a recording instead of a corpus, and every arm declares at the
+# checkpoint's training context, so neither can meet a sequence too long to boot.
+# Measured on the raw JSON line — the pessimistic direction, and the same figure
+# the census in the header quotes. It runs on a dry run too, because a refusal
+# this cheap should never have to wait for the drained window to be discovered.
+if [ "$MODE" = kldref ] && [ -f "$KLD_CORPUS" ]; then
+    read -r REC_NO REC_CHARS < <(awk '{ if (length($0) > best) { best = length($0); where = NR } } END { print where + 0, best + 0 }' "$KLD_CORPUS")
+    LIMIT_CHARS=$(( MAX_MODEL_LEN * 3 ))
+    if [ "$REC_CHARS" -gt "$LIMIT_CHARS" ]; then
+        echo "REFUSED: record $REC_NO of $KLD_CORPUS is $REC_CHARS chars, past the $LIMIT_CHARS a $MAX_MODEL_LEN-token window promises at the pessimistic chars/3 — raise FRANK_QWEN38_MAX_MODEL_LEN, and the container's declared --max-ctx with it, before booting the reference" >&2
+        exit 2
+    fi
+    echo "corpus fits: longest record $REC_CHARS chars (number $REC_NO) inside the $MAX_MODEL_LEN-token window (limit $LIMIT_CHARS chars)"
+fi
+
+if [ "$DRY" = 1 ]; then
+    echo "dry run: $MODE — the production unit is left running, nothing is mounted, nothing is written"
+    printf '%q\n' "${CMD[@]}"
+    exit 0
+fi
+
+# ---- gates that only a real run needs ---------------------------------------
+if [ "$MODE" = score ]; then
+    if [ ! -d "$KLD_REF_DIR" ]; then
+        echo "REFUSED: no KLD reference at $KLD_REF_DIR — record it first in a drained window: scripts/run-qwen38-bf16.sh kldref" >&2
+        exit 2
+    fi
+    if [ ! -f "$SCORE_RAD" ]; then
+        echo "REFUSED: no container to score at $SCORE_RAD" >&2
+        exit 2
+    fi
+    if [ -e "$SCORE_OUT" ]; then
+        echo "REFUSED: $SCORE_OUT already exists — one report per name; move the old one to trash before re-scoring" >&2
+        exit 2
+    fi
+    mkdir -p "$OUT_DIR"
+else
+    if [ ! -f "$RAD" ]; then
+        echo "REFUSED: no reference container at $RAD — build it first: scripts/build-qwen38-bf16-rad.sh" >&2
+        exit 2
+    fi
+    [ "$MODE" = kldref ] && [ ! -f "$KLD_CORPUS" ] && {
+        echo "REFUSED: no KLD corpus at $KLD_CORPUS — the record would be empty" >&2
+        exit 2
+    }
+fi
+if [ ! -f "$TEMPLATE" ]; then
+    echo "REFUSED: no chat template at $TEMPLATE" >&2
+    exit 2
+fi
+mkdir -p "$LOG_DIR"
+# The prod template, staged where the ro /models mount reaches it: a reference
+# recorded with the container's baked template scores a different tokenisation
+# than the arms served with froggeric's, and the KL would be measuring that.
+cp -f "$TEMPLATE" "$TPL" || {
+    echo "REFUSED: cannot stage the chat template at $TPL (the container reads it through /models)" >&2
+    exit 2
+}
+[ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
+[ "$MODE" = calib ]  && mkdir -p "$CALIB_DIR"
+
+restore() { sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true; }
+trap restore EXIT
+echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
+sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true
+podman rm -f "$C" >/dev/null 2>&1 || true
+
+"${CMD[@]}" 2>&1 | tee "$LOG"
+rc=${PIPESTATUS[0]}
+echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
+exit "$rc"

scripts/serve-frank-archon.sh

@@ -15,18 +15,21 @@
 # log states every budget before anything is allocated (weights vs KV
 # split, the expert host pool's coverage of the worst case, the PLE pin,
 # the P2P answer per pair) — read it after every resize.
-# Deployed shape (2026-10-19, radiance 1.8.1): THREE decode slots x 200,000
+# Deployed shape (2026-10-20, radiance 1.8.1): FOUR decode slots x 200,000 tokens over a
-# tokens, fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in
+# >= 700,000-token fp8 KV pool, 48 GiB/rank pinned expert pool, MTP depth 2, prefix cache
-# pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab
+# in pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV
-# and the KV pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair
+# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget
-# and the budget resolver derives it from headroom, the cache bound and the host
+# resolver derives it from headroom, the cache bound and the host pool, so the slab
-# pool, so the slab size and the pool's token count are LEDGER lines: after every
+# size and the pool's token count are LEDGER lines. After every boot read them, plus
-# boot read them out of the log and check the pool still backs 3 x 200,000.
+# the two lines that size a prefill step — "activation arena: ... lends ... while
+# steps are at most N tokens" (the arena levels the expert slab borrows) and
+# "prefill staging: ... a step of more than N tokens ..." (the chunk size below which
+# pooled experts are read synchronously) — and the cache's session backing.
 #
 # The root unit (scripts/frank-archon.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
+# LimitMEMLOCK=infinity) grants the pin budget: 2 x 48 GiB expert pools + the 47.7 GiB PLE
-# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
+# table + 8 GiB prefix host tier + the checkpoint store (slots x 112 MiB) — ~159 GiB of
-# pages, which is ALSO why this script refuses to start when ttm's
+# pinned pages, which is ALSO why this script refuses to start when ttm's
 # pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
 set -euo pipefail
 

scripts/serve-frank-dragoon.sh

@@ -136,10 +136,17 @@ fi
 #      The engine re-checks size+sha256 against that catalog at boot; it cannot
 #      fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
 #      stays unset — no outbound connection at boot),
-#   2. the distro XRT's three libraries, each mounted twice (where the image
+#   2. the distro XRT's three libraries, each mounted THREE times: where the
-#      looks for XRT, and at its own path — the libraries find each other there;
+#      image looks for XRT (/opt/xilinx/xrt/lib), at their own path (the
-#      an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
+#      libraries find each other there), and at /opt/xilinx/lib/<multiarch>
-#      container, which the vendor calls out explicitly),
+#      — the NPU child searches that third prefix too, and since the host's
+#      libxrt2 was reconfigured 2026-10-20 it is the ONLY prefix the child
+#      names: omitting it crashes the lane with `No such library
+#      '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the engine
+#      supervisor takes the dead lane down on the whole container (measured
+#      2026-10-21: 88-restart boot loop). Real files only — an /opt/xilinx/xrt
+#      of symlinks into the system dir dangles inside the container, which the
+#      vendor calls out explicitly),
 #   3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
 #      hang the machine while FCLK changes speed, and the engine refuses to
 #      start when it moves — it cannot read host sysfs from inside the
@@ -190,7 +197,9 @@ if [ -n "$NPU_MODELS" ]; then
             exit 1
         fi
         REAL="$(readlink -f "$XRT_DIR/$L")"
-        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
+        MULTIARCH="${XRT_DIR##*/}"
+        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro"
+                   -v "$REAL:/opt/xilinx/lib/$MULTIARCH/$L:ro")
     done
     HELD=0
     for D in "$SYSFS"/class/drm/card*/device; do

scripts/test-changed.sh

@@ -265,6 +265,13 @@ for path in "${CHANGED[@]}"; do
             add_test tests/voice_pack_unit.bats
             add_test tests/scripts_syntax.bats
             ;;
+        # The KLD corpus re-chunker: its seam is its own (the campaign's
+        # one-token-series invariant is asserted there), so a change to it does
+        # not land on the generic scripts/* fallback below.
+        scripts/rechunk-kld-corpus.py)
+            add_test tests/qwen38_rechunk.bats
+            add_test tests/scripts_syntax.bats
+            ;;
         scripts/*)
             add_test tests/scripts_syntax.bats
             ;;
@@ -293,6 +300,14 @@ for path in "${CHANGED[@]}"; do
             add_test tests/caddyfile.bats
             add_test tests/deployment_scoping.bats
             ;;
+        # The GPU exporter is a host process; its seam pins the collector, the
+        # user unit, the deployer function, the planner route, the Prometheus
+        # scrape jobs and the host-system GPU panels — a cross-file contract, so
+        # the whole seam rides on any of its own files.
+        scripts/gpu-exporter.py|scripts/gpu-exporter.service)
+            add_test tests/gpu_exporter.bats
+            add_test tests/deployment_scoping.bats
+            ;;
         dankclaw/udev/*)
             # Host-level udev rules are applied by manual sudo install + udevadm
             # trigger, not dispatched as a service; nothing here is bats-testable.

serverAppConfigs-docs/Agent/OMP Harness.md

@@ -52,28 +52,31 @@ Where the contract lives:
 The arithmetic (the coupled contract every endpoint's maxTokens comment
 encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
 endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
-follows is a **worked example** recomputed from the live files on 2026-10-18 —
+follows is a **worked example** recomputed from the live files on 2026-10-21 —
-a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
+a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
 maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
 
 1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
-   95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
+   95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
    bundle clamps the percentage to 1–99 and converts it to a token threshold.
 2. **Output budget** = capped at the target model's `maxTokens` — every role
    here compacts to frank-zealot, so the budget is its 16,384.
-3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
-   window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
+   dragoon session the fitting window is the ENGINE ceiling, not a slot: the
-   200,704. The registry's own comments design around an 85 % trigger (archon:
+   190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
-   170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
+   error plus the 45,056 request budget lands inside — that headroom is the
-   does hold. While `config.yml` and those comments disagree, treat 85 % as the
+   reason the ceiling moved to native on 2026-10-21. For a session compacting
-   design point and re-run this arithmetic after ANY window change — a 400 from
+   INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
-   the compaction target mid-conversation is the symptom.
+   does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
+   200,704. Treat 85 % as the design point until config.yml and the registry
+   comments agree, and re-run this arithmetic after ANY window change — a 400
+   from the compaction target mid-conversation is the symptom.
    (Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
    177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
    32,768 ≤ 200,704 zealot.)
 
 Procedure (recompute from the LIVE files every time — windows rotate with
-every swap; the block above is the 2026-10-18 state):
+every swap; the block above is the 2026-10-21 state):
 
 1. Read the `config.yml` compaction block + `models.yml` entries for source
    and target. Every endpoint's `contextWindow` comment carries its own
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
 as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
 dankclaw — the comment/config disagreed. The comments now name the real
 target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
-truth; fix comments in the same change.** A fresh instance of the same trap is
+truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
-open 2026-10-18: every `maxTokens` comment still says "compaction fires at
+disagreement is not academic: it is why long dragoon sessions kept drawing
-85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
+400s at the old 200,704 ceiling — the trigger fires late AND the est
+under-counts, and a window parked near the ceiling has no room for either.
 `maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
 input. Boot-shape OOM laddering is a different problem
 ([[Llama Cpp Serving Sizing]]).

serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

@@ -130,10 +130,19 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
 
 ## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
 
-- **Hard-fail keys** — the whole unit silently fails to generate
+- **Hard-fail keys abort the WHOLE BATCH, not one unit** (podman 5.7.0,
-  (`systemctl --user cat <unit>` shows "could not be found"): any
+  live 2026-10-21): one unsupported key — e.g. `HostAddr=` — makes the
-  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
+  generator log `unsupported key … in group 'Container'` plus
-  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
+  `processing encountered some errors` and generate NOTHING for that
+  manager. The running containers keep going as orphans, so the failure
+  first appears on the next deploy as `Unit <x>.service not found` for a
+  service that is visibly up. Other offenders: any dash-prefixed
+  `[Container]` option (`--shm-size=…`), `Command=` (use `Exec=`).
+  The seam `tests/quadlet_image_pins.bats` fails the build on any
+  generator-rejected key in any host's units.
+  Also: an image whose ENTRYPOINT is the binary takes FLAGS ONLY in
+  `Exec=` — repeating the binary makes it a positional arg
+  (`unexpected /bin/blackbox_exporter`, live 2026-10-21).
 - **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
   spaces into the healthcheck argv; a quoted probe
   `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
@@ -148,6 +157,16 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
   "unsupported key"`, `systemctl --user cat probe.service` (shows the
   generated ExecStart verbatim), then remove + reload.
 
+## 4a. Image pins (fleet law, 2026-10-21)
+
+Every registry image in a quadlet is digest-pinned (`tag@sha256:…`);
+locally built images (`localhost/*`, `franksdank/*`) are exempt — their
+build is the pin. A rolling tag lets a rebuild move under a running
+service with nobody approving it, and it makes a CVE audit ungradeable.
+Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
+tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
+version the config label carries in the comment beside it.
+
 ## 5. Rootless Quadlet log retrieval
 
 Step 0 — establish ownership: `systemctl --user status <name>.service`

serverAppConfigs-docs/Architecture/Stale Signal Audit.md

@@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** —
 docs, unit docstrings, and the repo's rule file (the most expensive place to
 be stale).
 
+## Three rules that outlive the incident that taught them
+
+* **A target can read down beside a healthy service.** Read `lastError` from
+  `/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the
+  protocol layer without `--metrics` (501) while `/health` answers 200 for weeks.
+* **Alert state lives where the tool says it does.** Prometheus keeps rule state in
+  process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is
+  not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a
+  host volume: silences + notification log, [[Alertmanager]]).
+* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production
+  unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape,
+  [[frank-zealot]]) beside an answering port means the metric measured a scratch boot;
+  the process shape can change between two censuses ten minutes apart.
+
 ## Order and verification
 
 1. Enumerate the surfaces (status board, metrics targets, alert rules, test

serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md

@@ -110,9 +110,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
 
 ## Recommended additions
 
-Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
+Quantized preferred. The iGPU's GTT aperture is not free RAM beside
-reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
+frank-dragoon — size every pick against *What actually fits beside
-2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
+frank-dragoon* below (the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
 all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
 `huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
 `huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
@@ -133,8 +133,34 @@ sha256 checks) or a plain `hf download` with the token in the environment —
 never on the command line.
 
 Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
-the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
+the 2.1 files already on disk), and any BF16 of #2/#4 — they double the disk
-them at GTT capacity, and they double the disk bill).
+bill, and see *What actually fits beside frank-dragoon* for why raw GTT size
+is not the real budget.
+
+## What actually fits beside frank-dragoon
+
+The 122 GB `/system_stats` figure is the GTT **aperture**, not free memory: on
+an APU the aperture and `MemAvailable` are the same physical RAM, and
+frank-dragoon pins tens of GiB of it. Measured 2026-10-21 with frank-dragoon
+live: `MemAvailable` 15.3 GiB, iGPU busy 1.75 GiB — ComfyUI itself idles at
+~10 MB, so the budget for a generation run is roughly what `free` says.
+
+Weights must land inside that budget with slack for compute buffers and
+activations (the mmproj-class lesson: price a model at ~1.5× its file set):
+
+| Pipeline (weights) | Fits beside frank-dragoon? |
+|---|---|
+| Qwen-Image 2.1 int8_convrot + w4a8 + bf16 VAE (~14.2 GB) — the two proven workflows | **No** — thrash-or-fail; do not queue it while the endpoint serves |
+| GGUF Q8_0 + w4a8 + small VAE (~13.3 GB) | No, same reason |
+| Any BF16 combination | No |
+| SDXL-Turbo / SD 1.5-class (~2–7 GB) | Yes, with real headroom |
+
+**The law: one process owns the memory — same rule as the 3090 bench law.**
+For a Qwen-Image render: stop and mask frank-dragoon
+(`sudo systemctl stop frank-dragoon` + `mask` — the mask stops
+`Restart=always` from firing mid-render), generate, unmask, verify
+`curl 192.168.0.143:7770/health`. The proven workflows above then run with
+the whole machine. Do not treat `/system_stats`'s 122 GB as an invitation.
 
 ## See Also
 

serverAppConfigs-docs/Dankclaw/ComfyUI.md

@@ -129,6 +129,12 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
 - Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
 - The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
 - The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
+- **The 122 GB `/system_stats` VRAM figure is the GTT aperture, not free
+  memory.** On the APU it is system RAM, and frank-dragoon pins most of it
+  (`MemAvailable` 15.3 GiB with the endpoint live, measured 2026-10-21). A
+  Qwen-Image-class render beside the running endpoint is thrash-or-fail; the
+  stop-mask-render-verify procedure and the fit table are in
+  [[ComfyUI Qwen-Image Workflows]] → What actually fits beside frank-dragoon.
 
 ## See Also
 

serverAppConfigs-docs/Dankclaw/OpenWebUI.md

@@ -34,15 +34,16 @@ https://github.com/open-webui/open-webui
 The picker is built from the `model` table in `webui.db` (one row per model)
 plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
 connection discovery — so the picker shows exactly what is seeded there.
-As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified
+Since 2026-10-21 the picker exposes **all three fleet models to every
-2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids
+account**: `frank-dragoon`, `frank-archon`, `frank-zealot` (per Frank: open
-`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
+`frank-archon` to everyone; the other two were already public). Proven with
-`shared/homepage/services.json`)*.
+a `role=user` token: `/api/models` lists all three ids and a chat turn to
+`frank-archon` answers end-to-end.
 
-| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
+| Connection (`openai.api_configs` idx) | `openai.api_base_urls` (live 2026-10-21) | Enabled | `model_ids` | Endpoint page |
 |---|---|---|---|---|
-| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
+| 0 | `http://host.containers.internal:7770/v1` | yes | `frank-dragoon` | [[frank-dragoon]] |
-| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
+| 1 | `http://192.168.0.235:7770/v1` | yes | `frank-archon` | [[frank-archon]] |
 | 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
 
 The URLs in the first column are the DB's recorded values and predate the 2026-08-22
@@ -50,9 +51,10 @@ bridge move; on this network they are wrong — see the reachability gotcha belo
 is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
 that day's payload swap (see [[frank-zealot]] for what was deployed and when).
 
-The `model` table carries exactly two decoration rows, `frank-zealot` and
+The `model` table carries three decoration rows — `frank-zealot`,
-`frank-dragoon`. Both the context length and the description live **inside
+`frank-dragoon`, `frank-archon` (200,000 context, added 2026-10-21).
-JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
+Both the context length and the description live **inside JSON columns** —
+`params` holds `{"context": {"length": …}}` and `meta` holds
 `{"description": …}`; there is no `context_length` or `description` column, so
 `sqlite3` one-liners against those names fail. `params.context.length` is a copy
 of the endpoint's per-slot window, so the authoritative number lives on the
@@ -380,6 +382,20 @@ neither.
   name: /selected model/i })`) and the welcome text "How can I help you
   today?"; that snapshot is also the cheapest proof the picker is populated.
 
+- **2026-10-21 — picker opened to all three models, for everyone.** Per
+  Frank: `frank-archon` accessible to everyone. Three-leg repair on
+  `webui.db` (snapshot `webui.db.bak-20261021` first): connection idx 1
+  `enable: true`, a `model` row `frank-archon` (params
+  `{"context":{"length":200000}}`, meta description from the endpoint card),
+  and an `access_grant` row `(resource_type='model', resource_id='frank-archon',
+  principal_id='*', 'read')` — `principal_id='*'` is the public grant, not a
+  role row. Verified as a plain `role=user` token: `/api/models` lists all
+  three ids, chat to `frank-archon` answers. Token-mint gotchas: the backend
+  package imports only with cwd `/app/backend`, and `WEBUI_SECRET_KEY` must
+  come from **PID 1's environ** (`podman exec` sessions do not inherit the
+  app process env; the `.webui_secret_key` file happens to equal it — use it
+  explicitly, `podman exec -e`).
+
 ## See Also
 
 - [[Dankclaw Stack]]

serverAppConfigs-docs/Dankserver/Plex.md

@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
 tags:
   - dankserver
   - plex
-timestamp: 2026-09-15
+timestamp: 2026-10-21
 ---
 
 # Plex
@@ -24,7 +24,7 @@ public traffic can be fronted by Caddy like every other vhost.
 | Item | Value |
 |------|-------|
 | Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
-| Image | `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (pinned; verified == the ghcr `latest` manifest digest at pin time) |
+| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
 | Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
 | Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
 | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |

serverAppConfigs-docs/Infrastructure/Alloy.md

@@ -3,70 +3,111 @@ type: reference
 description: "Alloy is Grafana's log and telemetry shipper."
 tags:
   - infrastructure
-timestamp: 2026-09-11
+timestamp: 2026-10-20
 ---
 
 # Alloy
 
-Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering **zero** log lines. See "Delivery state" below.
+Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life
+as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It
+forwards the host journal — including the rootless Quadlet containers' logs —
+to Loki. Unlike its observability siblings it runs as a **root system service on
+the host network**, because a rootless container cannot read the journal (see
+*Why root* below).
 
-**Host:** Dankclaw (one instance per host) — **Port:** 12345 (loopback) — **Docs:** https://grafana.com/docs/alloy/latest
+**Host:** Dankclaw (one instance per host) — **UI/health:** 127.0.0.1:12345
+(loopback) — **Docs:** https://grafana.com/docs/alloy/latest
 
 ## Facts
 
 | Item | Value |
 |------|-------|
-| Image | `franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck) |
+| Image | `localhost/franksdank/alloy:latest` — built rootless, then mirrored into the root image store by the deployer (a root build fails on DNS in this host's build container) |
-| Published | `127.0.0.1:12345` → 12345 (its own UI/health port) |
+| Unit | `dankclaw/observability/quadlet/alloy.container` → system `alloy.service`, installed at `/etc/containers/systemd/` (root manager) |
-| Unit | `dankclaw/observability/quadlet/alloy.container` → `alloy.service` |
+| Config | `shared/alloy/config.alloy` → `/opt/alloy/config/config.alloy` on every deploy |
-| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/config.alloy` on every deploy (`scripts/deploy-remote.sh:281-284`) |
+| Network | `host` — required so it can reach Loki's `127.0.0.1:3100` publish |
-| Inputs | host journald socket, podman container logs, `/var/log` |
+| UI | `--server.http.listen-addr=127.0.0.1:12345` (loopback; nothing off-host reaches it) |
-| Output | `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) |
+| Inputs | host journal: `system.journal` + `user-*.journal` (root reads both) |
-| Successor to | Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/) |
+| Output | `http://127.0.0.1:3100/loki/api/v1/push` |
-| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alloy.container:58-63`); installed at `~/.config/containers/systemd/alloy.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active alloy` 2026-10-18) |
+| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=multi-user.target`; boot persistence is the `[Install]` section (generated system units cannot be `enable`d here) |
-| Delivery | **Configured and healthy, currently delivering zero data** — see "Delivery state" |
+| Successor to | Promtail (EOL 2026-03-02) |
+| Deploy status | root cutover built and tested in-repo (2026-10-20); the **live** instance is still the old rootless unit until `./scripts/deploy.sh dankclaw --service alloy` runs |
 
-## Why Alloy and not Promtail
+## Why root (the load-bearing decision)
 
-Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (`git log -S promtail` over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — `dankclaw/observability/quadlet/alloy.container:6-7`).
+A rootless Alloy delivered **zero** lines, silently and with a green health
+check. Two independent causes, both proven on the host 2026-10-20:
+
+1. **The journal is not readable by a rootless container.** The files are
+   `root:systemd-journal 0640` (`group:adm`, `other::---`). Even bind-mounted, a
+   rootless container has no supplementary groups, so opening them returns
+   `DENIED`. `loki.source.journal` then sees an empty view and the counter just
+   sits at 0 — no error logged.
+2. **Container logs live in the *user* journal, not `ctr.log`.** The rootless
+   Quadlet containers log via the journald driver into `user-1000.journal`; the
+   old file globs (`/var/lib/containers/.../ctr.log`, `/var/log/containers/*.log`)
+   point at a root-storage layout that does not exist here.
+
+**Root fixes both at once.** As root the container opens every journal file
+under the machine-id directory — `system.journal` *and* `user-1000.journal` —
+so a single `loki.source.journal` carries host services and the rootless
+containers' logs together. Validated with a throwaway root probe before writing
+the unit: **31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero
+drops**, and Loki's label set went from empty to populated.
+
+### The two mounts that make it work
+
+- **`/etc/machine-id` (ro) — load-bearing.** `sd_journal` locates the journal
+  directory by the *local* machine id. Without it the container opens nothing and
+  the journal source reads exactly 0 lines — silently, health green. This single
+  mount is the whole difference between a working and an empty pipeline; do not
+  remove it. (A root probe without it reads 0; with it, 31,825.)
+- **`Network=host`.** Loki publishes `127.0.0.1:3100` (loopback only). No bridge
+  container — rootless or root — can reach the host's loopback, so Alloy must
+  share the host network to push to `127.0.0.1:3100`. Consequence: the UI binds
+  `127.0.0.1` directly (no `PublishPort`), so it stays loopback-only as before.
 
 ## Configuration
 
-`shared/alloy/config.alloy` is the source of truth:
+`shared/alloy/config.alloy` is the source of truth — one `loki.source.journal`
-
+(labelled `job="systemd-journal"`) forwarding to the loopback `loki.write`
-- Reads journald logs from the host's `/run/systemd/journal/socket`, labelled `job="systemd-journal"` (`shared/alloy/config.alloy:20-26`).
+endpoint. There is deliberately **no** container-file source: container stdout is
-- Reads container logs from two globs: `/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log` (`job="podman"`) and `/var/log/containers/*.log` (`job="containers"`) (`config.alloy:29-35`).
+in the journal.
-- Forwards everything to `http://loki:3100/loki/api/v1/push` by container name on `dankclaw_default` (`config.alloy:13-17`).
 
 ## Volumes
 
 | Container path | Host path |
 |----------------|-----------|
-| `/etc/alloy` | `/opt/alloy/config` |
+| `/etc/alloy` | `/opt/alloy/config` (ro) |
-| `/run/systemd/journal/socket` | host journald socket (read-only) |
+| `/etc/machine-id` | host machine id (ro) — **required** |
-| `/var/log` | host logs (read-only) |
+| `/run/systemd/journal/socket` | host journald socket (ro) |
-| `/var/lib/containers` | podman storage (read-only) |
+| `/var/log/journal` | persistent journal files (ro) |
+| `/run/log/journal` | runtime journal files (ro) |
 
-## The UI port is published on purpose
+## The UI stays on loopback
 
-Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`).
+Alloy 1.x refuses to configure the `ui` service from a config block, so the UI
-
+address is set on the command line instead. Under the old rootless unit that
-## Delivery state (2026-10-18): healthy, shipping nothing
+meant `0.0.0.0` inside the container plus a published `127.0.0.1:12345`; under
-
+`Network=host` the listen-addr is `127.0.0.1:12345` directly — the host's
-The unit is `active`, the container reports `healthy`, the log says `Alloy is ready`, and the pipeline carries no data at all. Measured on 2026-10-18:
+loopback, reachable from the host and from nothing off-host. Alloy has no
-
+authentication, which is exactly why it is never published wider. The homepage
-- `curl -s http://127.0.0.1:12345/metrics` → `loki_source_journal_target_lines_total{component_id="loki.source.journal.journal"} 0` and `loki_source_file_files_active_total{component_id="loki.source.file.containers"} 0`. The journal source logs no error; the counter simply never moves. The journal files it would have to read through the `/var/log` bind are `root:systemd-journal 0640` with an ACL (`ls -l /var/log/journal/64a287d7…/` 2026-10-18).
+card and the deploy health gate both dial `http://127.0.0.1:12345/-/healthy`.
-- Both file globs were refused at start (`journalctl --user -u alloy`, 2026-10-18T08:25:27Z), verbatim: `failed to create source, skipping … component_id=loki.source.file.containers error="failed to tail file, stat failed: stat /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log: permission denied"` and the same message for `/var/log/containers/*.log: no such file or directory`.
-- Downstream that means Loki holds zero streams — `GET /loki/api/v1/labels` returns `{"status":"success"}` with no `data` — see [[Loki]].
-
-No fix is claimed here: the unit, the config and the push endpoint are all correct as written, and the gap is source-side access (and a Kubernetes-style glob that does not exist on this host). Anyone closing it should re-check the two counters above, not the health status — health is exactly what hides this failure mode.
 
 ## Gotchas
 
-- The journald socket mount is required for journal scraping.
+- **No `/etc/machine-id` mount ⇒ silent zero.** The health check still passes.
-- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host.
+  This is the failure mode that hid the broken pipeline; check
-- One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
+  `loki_source_journal_target_lines_total` and the Loki label query, never
-- The unit `Wants=`/`After=` `loki.service`: without a reachable Loki the write client fails and the pipeline backs up.
+  `/healthy`.
+- **`After=loki.service` is gone.** Alloy is on the system manager; Loki is a
+  rootless user unit — a system unit cannot order against a user unit. Alloy
+  retries its push until Loki answers, so ordering is unnecessary.
+- **The image must be in the ROOT store.** The deployer builds rootless and
+  mirrors (`mirror_image_to_root`); a `Pull=never` root quadlet with only a
+  rootless tag fails at create.
+- One Alloy per host; do not run a single instance that tries to reach both
+  hosts' logs.
 
 ## See Also
 

serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md

@@ -81,7 +81,7 @@ Two provisioning rules, each verified live by an outage:
    appears. Create the Application with `provider_id` set (the UI does this
    automatically; raw API/ORM drives must do it explicitly).
 
-## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)
+## OAuth2 relying-party quirks (this fork; corrected live 2026-10-20)
 
 The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
 verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
@@ -100,23 +100,32 @@ implementation diverges from upstream in ways that bit us; all verified live:
 - **Provider resolution is by `client_id` query param** on the global
   `/application/o/authorize/` route; per-slug authorize paths do not exist in
   this build. Token/userinfo live at `/application/o/token/` and
-  `/application/o/userinfo/?api=1`.
+  `/application/o/userinfo/` — do **not** append `?api=1` to a Grafana API URL.
-- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
+- **NULL `authorization_flow` → authorize 500, `AttributeError: 'NoneType'
-  `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
+  object has no attribute 'slug'`.** A provider created via raw `ak shell`
-  `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
+  `create()` leaves `authorization_flow` unset; the authorize view then crashes
-  becomes the *expression source* instead of the context; every call raises a
+  building the flow planner (`flows/planner.py`) and returns "Server Error —
-  swallowed `PropertyMappingExpressionException` and the claim is skipped.
+  Powered by authentik" before any claim is built. This was previously
-  Scope-grant filtering has the same class of bug: `ScopeMapping` has no
+  mis-attributed to a broken evaluator; the crash is the missing flow. Fix: set
-  provider FK (linkage lives in the `ProviderPropertyMapping` through model)
+  `authorization_flow = default-provider-authorization-implicit-consent` (the
-  so `filter(provider=...)` always yields an empty queryset and tokens asked
+  flow every other provider in this instance already uses).
-  to carry scopes get zero. Net effect: userinfo/introspection answers with
+- **A scope mapping that returns a non-dict emits no claim.** authentik merges a
-  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
+  granted scope mapping's evaluated result into the token **as a mapping**: a
-  build until the evaluator is patched or upgraded. Clients consuming this
+  dict contributes its keys as claims; a bare string is keyed under the
-  provider must key identity off `sub`.
+  mapping's NAME, so nothing usable appears. `grafana-email-scope` returned the
-- Scope mappings for this provider exist — `grafana-email-scope` and
+  bare string `user.email`, so no `email` claim landed — the evaluator was
-  `grafana-profile-scope` are both rows in `authentik_core_propertymapping`
+  never broken (the dict-valued `grafana-profile-scope` mapping has always
-  (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
+  delivered, and the stock OpenID mappings return dicts). Fix:
-  the evaluator bug is fixed; no config change is needed then.
+  `return {"email": user.email, "email_verified": True}`. With no email, Grafana
+  v13 falls back to GET `<api_url>/emails`; authentik 404s it → "Login failed:
+  InternalError".
+- **Grafana reads identity from the standard userinfo endpoint** (`API_URL` with
+  no `?api=1`); no `USE_ID_TOKEN` is needed once the `email` claim exists.
+- **Net (2026-10-20):** the evaluator is fine; email/profile claims reach RP
+  clients. Both IdP fixes are idempotent in
+  `scripts/authentik-grafana-sso-fix.sh` (re-run after an ak-shell re-creation of
+  the provider or a blueprint import, which revert them). Grafana's half plus the
+  verified outcome (Viewer, `login = <sub hex>`, email present) are in [[Grafana]].
 
 ## LDAP provider objects (provisioned via `ak shell` ORM)
 

serverAppConfigs-docs/Infrastructure/Grafana.md

@@ -4,7 +4,7 @@ description: "Grafana visualizes the metrics that Prometheus collects from every
 tags:
   - infrastructure
   - grafana
-timestamp: 2026-09-11
+timestamp: 2026-10-20
 ---
 
 # Grafana
@@ -64,19 +64,28 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
 - Change the default admin credentials at first boot (env or UI).
 - Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
 - **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
-  no provider token (`token=None`), so no identity header ever reaches Grafana;
+  no provider token (`token=None`), so no identity header reaches Grafana; the
-  RP mode is topology-independent. The full verified contract (provider pk 9,
+  login is Grafana's own "Sign in with Authentik" button against the `grafana`
-  client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths,
+  confidential-client provider. **Verified end-to-end 2026-10-20** after four
-  Basic client auth, form-encoded token body, and the fork's userinfo-claims
+  independent breakages had made it fail. The two Authentik-side causes (a NULL
-  bug that leaves identity at the `sub` hash only) is in
+  `authorization_flow` → authorize 500; an email scope mapping that returned a
-  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana:
+  bare string so no `email` claim was emitted) and the full wire contract live in
-  The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`
+  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks"; the IdP half is
-  `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
+  re-applied by `scripts/authentik-grafana-sso-fix.sh` (idempotent — re-run after
-  users land as Viewer with `login = <sub hex>`; no email/groups/role claims
+  an ak-shell re-creation of the provider or a blueprint import, both of which
-  reach Grafana until the evaluator bug is fixed — do not write role mappings
+  revert it silently). The Grafana half is the unit's own env (`grafana.container`):
-  that depend on those claims. The local admin account stays the only Admin.
+  `SCOPES="openid email profile"` (a GitLab-style `user:email` scope crashes the
-  (The Viewer-default outcome and the `<sub hex>` login are not readable from
+  authorize view), the standard userinfo `API_URL` with **no `?api=1`**, the
-  the host without the database or an admin password. *(unverified 2026-10-18)*)
+  attribute paths `email`/`sub`/`preferred_username`, and `ALLOW_SIGN_UP=true`
+  (safe: the guard fronts the callback). Grafana v13 only GETs `<api_url>/emails`
+  when the token carries no `email` claim — so it is the email claim itself (an
+  IdP fix, above) that makes the login land, not any Grafana toggle.
+- **What a new login becomes:** a Viewer in org 1 (`ALLOW_ASSIGN_ORG=true`,
+  `ORG_ID=1`, no role-attribute mapping), with `login = <sub hex>` but the real
+  `email`. The local admin account stays the only Admin. The landing that the
+  edge-only 13-sso spec never checked is now asserted by
+  `tests/e2e/specs/20-grafana-sso.spec.ts` (RP round-trip → `/api/user` 200 with
+  an email).
 - **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
   file is a gluetun env file and it carries `HOME=/root`; systemd imports it
   for the service, podman then looks for its configuration under `/root`

serverAppConfigs-docs/Infrastructure/Observability Provisioning.md

@@ -3,7 +3,7 @@ type: reference
 title: Observability Provisioning
 description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
 tags: [observability, prometheus, grafana, provisioning, deployment]
-timestamp: 2026-10-05T00:00:00-07:00
+timestamp: 2026-10-21T00:00:00-07:00
 ---
 
 # Observability Provisioning
@@ -20,6 +20,8 @@ committed.
 | `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
 | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
 | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
+| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`); the listen bind lives on the unit's `Exec` |
 
 Targets and datasources reference **container names** (`grafana:3000`
 `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
 reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
 the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
 (`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
-HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
+HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
-[[Stale Signal Audit]]).
+deployed 2026-10-21; see [[Stale Signal Audit]]).
 
 ## The deploy path
 
@@ -43,8 +45,9 @@ from `deploy_observability`:
 - `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
 - `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
 - `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
-- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
+- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
-- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
+- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
 
 `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
 `alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
@@ -82,11 +85,14 @@ reports 4 alerting + 2 recording. Rules reload with the prometheus service.
 the backend (`http://prometheus:9090`), not loopback.
 
 **Add a dashboard** — drop the JSON into
-`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
+`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
-unified storage, not (only) the SQLite table — grep the API, not the DB, when
+the next `--service grafana` deploy provisions it. This is a repaired law, not
-verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
+a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
-2026-10-18: the storage-backend claim needs an admin password to confirm
+`containers` board — merged, tested, documented as "working" — had **never
-against `grafana.db` and the API side by side)*
+been provisioned**; the other boards existed on disk only because someone had
+hand-copied them once. A repo JSON is not a live dashboard until the copier
+carries it, and `deploy.sh` cannot see the gap because it only reports the
+copy it performed.
 
 ## Verify through the APIs, not the filesystem
 
@@ -95,90 +101,32 @@ against `grafana.db` and the API side by side)*
 curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
 curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
   python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
-# grafana: datasource + dashboard provisioned
+# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
-curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
+# account is the Authentik RP login). The proven surface is a real browser
+# session: warm the Anubis challenge on the vhost root, then hit
+# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
+# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
 ```
 
 The two Prometheus queries were run on 2026-10-18 and are what every number on
 this page comes from: 4 alerting + 2 recording rules, and `up` for
 `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
-`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
+`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
-listed in [[Prometheus]]. The Grafana query needs the local admin password and
+that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
-was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
+2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
-proof of installation only — see the next paragraph.
+filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
+only — see *Add a dashboard* for why a repo JSON is not a live board.
 
 Health answers from a stale process and provisioned resources can hide in
 unified storage — a file existing on disk is not proof the running server
 loaded it. See [[Stale Signal Audit]] for the never-green-target decision
 table (refused = outage, 404/HTML = config defect).
 
-## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
-
-Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
-loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
-unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
-via `host.containers.internal`) → signal-cli, addressed by number
-`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
-2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
-`curl -s http://192.168.0.143:18093/health` → `ok`, and
-`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
-lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
-`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
-structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
-the blackbox exporter. See [[Prometheus]] → Targets.
-
-**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
-`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
-the oldest sample, 2026-10-11, is already 0 — while the service itself answered
-`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
-is started with the flag, so the scrape was refused at the protocol layer (the
-flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
-stayed firing:
-the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
--u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
-on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
-metrics opt-in landed that day; the target
-reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
-How many payload generations it spanned is not readable from any host
-artifact. *(unverified 2026-10-18)*
-
-Two laws fall out of it. A target can read down beside a healthy service, so
-read `lastError` from `/api/v1/targets` before believing an outage. And rule
-state lives in Prometheus' process, not on disk — a restart re-stamps
-`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
-evidence of a fresh fault. (Alertmanager by contrast **does** persist:
-`--storage.path=/alertmanager` on a host volume keeps silences and the
-notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
-`/opt/alertmanager/data`.)
-
-Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
-metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
-([[frank-dragoon]]), which is why
-`frank-dragoon` has always been green. The active-alert count is a
-moment-in-time value and should never be documented as a constant:
-`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
-`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
-minutes later, and was empty again by 11:56Z; the relay logged the matching
-firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
-
-**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
-production `frank-zealot` root unit was stopped **on purpose** for a benchmark
-window on the 3090 — the repo Benchmarking Policy says stop the production
-service before benching — and `:7771` answered from a manually booted candidate
-arm (`systemctl is-active frank-zealot` → `failed` after
-`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
-`TimeoutStopSec`, the known shutdown shape for these engines). So
-`up{job="frank-zealot"} == 1` that day measured the trial, not the production
-shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
-and it is the repo contract.
-The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
-reinstalls and starts it, restoring boot persistence and self-healing. Never
-quote a candidate arm's flags or VRAM as fleet state — the process and its shape
-changed between two censuses ten minutes apart.
-
 ## Related
 
 - [[Observability]] — stack topology and the alert delivery chain
-- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
+  down-beside-healthy / alert-state-persistence / candidate-arm rules
 - [[Quadlet Lifecycle]] — the units that host these services
 - [[Deploy State Triage]] — how the planner selects these services

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -2,18 +2,18 @@
 type: reference
 title: "Observability on Dankclaw"
 description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
-tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
+tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
-timestamp: 2026-09-11
+timestamp: 2026-10-20
 ---
 
 # Observability on Dankclaw
 
 ## Summary
 
-Six containers and one host-side process: Prometheus (metrics), Grafana
+Seven containers and two host-side processes: Prometheus (metrics), Grafana
 (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
-metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
+metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
-carrying every firing group to Signal. All of it is Quadlet-managed,
+plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
 boot-persistent and loopback-only, deployed by
 `scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
 end to end; the log half is configured, healthy, and carries no data.
@@ -34,6 +34,8 @@ that proves it today.
 | Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
 | Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
 | node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
+| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
+| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
 | Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
 
 Every unit lives in `dankclaw/observability/quadlet/`, carries
@@ -44,9 +46,11 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
 the same day, and `homepage-status.timer` — the root unit that probes all of
 these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
 the probe registry `shared/homepage/services.json` to `/srv/homepage`).
-Dankserver runs one member of this family, node-exporter, as its own unit
+Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
-(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
+(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
-`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
+`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
+cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
+`gpu-exporter` (a user unit installed the same way as `signal-relay`).
 
 ## Exposure
 
@@ -63,10 +67,26 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
 in the host network namespace, and it binds exactly two addresses rather than
 the wildcard — see [[Node-Exporter]].
 
-The relay's bind is the one address a bridge container can reach on the host:
+**Same-host reachability from a bridge container is measured, not inferred**
-`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
+(2026-10-21, from inside the Prometheus container): `host.containers.internal`
-and the podman bridge subnet `10.89.0.0/16` before the DROP
+(the podman gateway) reaches host services whether they bind host-net or are
-(`scripts/deploy-remote.sh:255-262`).
+published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
+— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
+every same-host Prometheus target uses the gateway and only cross-host targets
+(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
+relay at `192.168.0.143:18093` is reached from alertmanager's network only
+while `install_signal_relay`'s INPUT rules are in force: loopback, the host
+itself and the podman bridge subnet `10.89.0.0/16` accepted **above** a
+trailing blanket DROP. The law the poisoned host taught (2026-10-21):
+`iptables -I` prepends, so inserting the DROP last lands it first — the old
+install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the
+deployer's health gate failed beside a LISTENing socket, and the drain
+aborted twice before `nft list ruleset` named the cause. The install now
+rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head,
+DROP appended); verify any change to it with the sandboxed bats case, not
+with `-C` greps — `-C` answers existence, never order. Never point a scrape
+target at a host LAN IP from its own host on the strength of one working
+example; dial the gateway and probe.
 
 ## Deployment
 
@@ -83,34 +103,74 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
 [[Observability Provisioning]] for what is provisioned as code and how to
 verify it through the APIs.
 
-## Current state (2026-10-18)
+## Current state (2026-10-21)
 
 * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
-  eight up targets; the four downs are structural (the `comfyui` and `services`
+  every job up — the three LLM endpoints (frank-archon on `vllm:*`,
-  HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set:
+  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
-  4 alert rules + 2 recording rules (`/api/v1/rules`, matching
+  both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
-  `shared/prometheus/rules/llm-fleet.yml`).
+  the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
+  downs are retired — see [[Prometheus]] → Targets.
+  Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
+  (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
+  alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
+  time-to-first-token), each keyed only to the namespace its runtime exports.
 * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
   relay, and `journalctl --user -u signal-relay` records successful
   `sent to +12062476336:` lines for every rule in the set.
-* **Dashboards: working.** One provisioned Prometheus datasource (uid
+* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
-  `prometheus`) and the `llm-fleet` dashboard, both installed from
+  datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
-  `shared/grafana/provisioning/`.
+  `llm-fleet`, `host-system`, `containers`, `observability` and
-* **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both
+  `logs-overview` boards. Until 2026-10-21 the copier carried only
-  file globs refused) and Loki therefore holds zero streams — `/opt/loki/data`
+  `llm-fleet.json` by name and the `containers` board had never been
-  is a few hundred KB of WAL/compactor bookkeeping with **no chunk files**
+  provisioned; the deployer now globs the directory. The Loki datasource is
-  (8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns
+  how you read logs: Loki has no web UI of its own, so Grafana Explore is the
-  no label names.
+  human surface (the guarded `loki.franksdank.cloud` vhost serves only the
-  See [[Alloy]] → Delivery state and [[Loki]] → Contents.
+  API). Verify method and the copier law: [[Observability Provisioning]].
-* **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the
+* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
-  structural downs have no real uptime signal), no Alertmanager vhost.
+  after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
+  per-container CPU/memory here, and on this rootless-Podman host it works on one
+  condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
+  namespace, so without that flag cadvisor sees only its own cgroup root and
+  exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
+  names nothing itself — the libpod socket resolved only 2 of 34 rootless
+  containers — but the quadlet unit name is carried in the cgroup path
+  (`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
+  `container` label. Network and filesystem are NOT exported for these containers
+  (no netns/classid attribution under the host-PID layout), so the dashboard
+  graphs only CPU, memory and OOM and does not fake the rest.
+* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
+  `deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
+  host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
+  own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
+  without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
+  `mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
+  emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
+  °C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
+  job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
+  `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
+  the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
+  node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
+* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
+  pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
+  the container-log globs pointed at a root-storage layout that does not exist
+  here) — fixed 2026-10-20 by making Alloy a **root system service on
+  `Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
+  but the stream carried only `job`/`service_name`, so every per-container
+  view was structurally empty: Loki metrics queries group only by INDEXED
+  stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
+  label (plus a `host` label) at ingestion — `loki.process` in
+  `shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
+  purpose-test law in [[Observability Provisioning]].
 
 ## Citations
 
-- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
+- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
 - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
 - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
+- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
 - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
 - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
+- `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
 - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
 - Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`

serverAppConfigs-docs/Infrastructure/Prometheus.md

@@ -42,29 +42,29 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 
 The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
 
-**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
+**No job reads down structurally any more** (the three permanent-reds were
+retired 2026-10-21; census re-run after each deploy):
 
-| Job | Target | `health` | `lastError`, verbatim |
+| Job | Target | how |
-|-----|--------|----------|------------------------|
+|---|---|---|
-| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
+| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
-| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
+| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
-| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
 
-Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
+Scraping a web root is not a health check: an HTML page fails the scrape
+protocol (`unsupported Content-Type`), which is what kept those targets red
+beside healthy services. The classification rule and the honesty law live in
+[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
+netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on loopback and
+the LAN, and Prometheus scrapes it at `host.containers.internal` — traffic to
+that gateway lands on the host's loopback listener. The gateway address is
+not itself bindable from the host netns.
+Plain targets stay `host:port`, never URLs — the probe job's URL targets are
+the documented exception, relabeled onto the prober.
 
 **`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
 
 Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
 
-The remedy for a probe-shaped question is the **blackbox exporter**; the
-classification rule and the honesty law for it live in [[Stale Signal Audit]].
-Until the exporter is deployed, these three stay red for a structural reason:
-deleting them would
-hide a real service, and fixing them by publishing databases or enabling metrics
-would trade a wrong signal for a real exposure. Targets are `host:port`, never
-URLs.
-
 ## Volumes
 
 | Container path | Host path |

serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md

@@ -239,45 +239,92 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp
 sequence stops producing entirely and then re-prefills from token 0, which is a
 stall, not sludge.
 
-Knobs, in the order worth trying them (radiance GUIDE §4.5):
+**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are
+looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is
+the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) /
+(steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with
+decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**,
+with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only
+14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so
+a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is
+worth ~10 % of one step — not a lever.
 
-- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N
+**Four traps in the PromQL above.** Learn them before trusting a bucket:
-  steps carry decode rows only and admit nothing. Decoders keep their cadence; the
+- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e
-  prompt pays that in TTFT. Counts as `engine_steps_held_total`.
+  histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption
-- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work:
+  is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the
-  `auto` (default) reserves a row plus the draft window per decode row behind it and
+  *instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the
-  an eighth of the budget per waiting prefill/request, `off` lets the chunk take the
+  endpoint never stalls anyone".
-  whole step (admission never runs while it does), `N` pins the share. Never under
+- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is
-  the prefill stager's smallest pass — the startup line names it, and below it every
+  `decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one
-  routed layer reads its pooled experts synchronously, which costs the prefill far
+  scrape in eight — use step ≤ 30 s for a bucket table.
-  more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
+- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is
-- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own
+  one shape (`up`, or the first sample of a counter that only the current build exports) before
-  curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE
+  ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse"
-  at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of
+  turned out to belong entirely to the older one.
-  magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses
+- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query
-  expert residency to the arena.
+  above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode.
-- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every
-  shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena
-  back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing
-  at short prompts, a large TTFT gain at the far end of the window.
 
-Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
+Three instruments the PromQL above cannot reach:
-expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the
-decode rows inside that step also read experts over the link — a smaller shared budget helps twice.
-And **re-check
-`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
-pool are the same memory, and the failure looks like turn two taking as long as turn
-one.
 
-Availability is a version question, not a config question: `--busy-prefill-chunk` and
+- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
-`--prefill-interleave` are absent from older pins — prove it on the running container
+  `token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine /
-(`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`), never from the vendored
+  prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on
-tree, and check the source clone's depth before dating a flag from `git log -S`
+  the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry,
-(a `--depth 1` clone reports its own tip for every search).
+  and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call:
+  configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes.
+- **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid
+  (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so
+  cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its
+  prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high
+  prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` -
+  exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner:
+  [[frank-archon]].
+- **Bucket the arriving request too** — `histogram_quantile` over
+  `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees
+  sequences that are already running; the one waiting for its first token pays the same steps and
+  shows in no decode number.
 
-Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 =
+Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee):
-interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict
+
-for this fleet's shape are that page's.
+- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode
+  rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their
+  cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps.
+  Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone.
+- `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed
+  setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth
+  of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows
+  alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from
+  `engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill
+  stager's smallest pass — the largest arena level that lends, plus one, named on the startup line
+  `prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it
+  every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four
+  decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the
+  chunk blindly buys latency in both directions.
+- `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then
+  carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own
+  measurement says a span starts to win), while **every other step is planned at the shared
+  budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk
+  knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV
+  pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first.
+- `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the
+  arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses
+  decode — [[frank-archon]]).
+
+Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is
+reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a
+quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller
+slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the
+arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two
+taking exactly as long as turn one.
+
+Availability is a version question, not a config question: `--busy-prefill-chunk`,
+`--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running
+container (`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s
+`configuration` block, never from a vendored tree, and check a source clone's depth before dating a
+flag from `git log -S` (a `--depth 1` clone reports its own tip for every search).
+
+Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms.
 
 ## Related
 

serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md

@@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
 
 `--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
 reserves the registry price, not the actual pool. Raising the pool consumes
-real free VRAM without growing the arena; the registry over-prices the pool
+real free VRAM without growing the arena; the registry OVER-PRICES the pool
-(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
+(era G measured ~14 % under the planned B/token — the gap is the lesson, not
-**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
+the number; **never reuse a B/token ratio across geometries**), and there is a
-block-alignment loss when converting a token target to bytes). The ask lands
+~4 MB block-alignment loss to subtract when converting a token target to
-block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
+bytes. The ask lands block-aligned: verify against the engine's own
-`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
+`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot.
 
 **Driver-reported free VRAM after hours of traffic is NOT allocatable
 arena.** It is the estimator gap: the registry over-prices KV and charges
@@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
 **KV accounting — pin the pool BYTES, floor-sized:** set
 `--kv-cache-memory-bytes` PER RANK to the contract floor
 (`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
-come out of the arena/headroom and buy nothing. Measured B/token on this
+come out of the arena/headroom and buy nothing. The B/token rate is a property
-hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
+of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**;
-dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
+the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape.
-7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
+Read your live rate from the boot ledger every time; the decomposition (a
-planner's KV reserve is for ONE max-model-len request.
+dtype-independent mamba/side stream rides on top of the attention KV) is why
+the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve
+is for ONE max-model-len request.
 
 **Parallelism ceilings you cannot flag past:**
 
@@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness
 (`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
 + engine flags in the log.
 
-**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
-
-| shape | util | graphs | arena | result |
-|---|---|---|---|---|
-| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
-| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
-| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
-| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
-
-bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
-at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
-~207k per context max.
 
 ## 7. Arena audit procedure (when asked for "more experts")
 

serverAppConfigs-docs/LLM/frank-archon.md

@@ -88,25 +88,30 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
 | Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
 |---|---|
 | TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
-| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
+| Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
 | KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
 | VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
-| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
+| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
 | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
 | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
 | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
 | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
-| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
+| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
-| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
+| TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
 | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
 
-Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`):
+Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
-the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**,
+the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
-`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages
+`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
-**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the
+The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
-2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots
+sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
-(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live
+per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
-(≤ 4,194,304 px, 16,384 patches/pass).
+`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
+host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
+(= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
+(≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
+directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
+the bigger cache, paid over PCIe when those experts are hit.
 
 The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
 anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
@@ -145,6 +150,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
 
 | Window | Payload | Runtime | Shape | Replaced because | Rollback |
 |---|---|---|---|---|---|
+| 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
 | 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
 | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
 | 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
@@ -211,27 +217,44 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
   buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
   the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
-- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
+- **At 1.8.1 the split belongs to the resolver, and the source fixes its order.**
-  The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and
+  `core/mem/vram_budget.cpp` claims the card bottom-up (`capacity → already held → drift → arena →
-  `--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
+  staging → queues → prefetch → headroom = claimable`), takes the **static weights off the top as a
-  sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
+  floor, not a share**, and divides the `elastic` remainder in three lines (146-156):
-  hand. Three boots of evidence:
 
-  | shape | demand | cache given | backing |
+  ```cpp
+  exp = std::min(b.expert_floor, b.elastic);
+  kv  = std::min(kv_cap, std::max(b.elastic - exp, kv_one));
+  exp = std::min(std::max(b.expert_ceiling, b.expert_floor), b.elastic - kv);
+  ```
+
+  `kv_cap` is `KVManager::ceiling()` = `state + Σ_paged max_seqs × ceil(max_ctx / block) × bytes`
+  (`core/mem/kv.cpp:266`), so `--max-num-seqs` and `--max-model-len` set the cache's **demand**, never
+  its funding. The funding is `elastic − expert_floor`, and the floor is
+  `min(plane + spare, plane + spare + 2·unit_max − host_room)` (`core/place/planner.cpp:100-104`, with
+  `host_room = host_pool − file_stage_reserve` at `core/engine_bringup.cpp:1418-1422`). **That is the
+  cache's lever: every MiB added to `--host-pool-mib` drops the expert floor one-for-one and the cache
+  takes it** — minus `stage`, the prefill stager's ring (two layers of the experts left off the card),
+  which is charged back out of the cache whenever the cache lends (`vram_budget.cpp:193-203`). Boots of
+  evidence, all on 1.8.1:
+
+  | shape | demand | cache given | engine's `backs %` |
   |---|---|---|---|
-  | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
+  | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
   | 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
-  | 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
+  | 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
+  | 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
 
-  So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
+  The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
-  never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
+  `ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
-  slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
+  ledger line `kv: the pool backs N % … the cache can grow to … N tokens`, never arithmetic.** To buy
-  grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
+  window: raise `--host-pool-mib` (cost: pinned host RAM, and experts evicted to host cross PCIe —
-  `--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
+  `experts-from-host` was 5.99 GiB a rank at 44,928 and is 8.99 at 49,152) or cut the per-token cost
-  hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
+  (`fp8` KV). Raising slots raises demand **and** costs 10 × 111.98 MiB of pinned checkpoint-store RAM
-  era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
+  per slot. Under-sizing the pool the other way has always been a boot failure (36,864 refused a boot in
-  the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
+  the fp8 era: the host cannot hold what the cards will not). Behavioural check on the live shape:
-  `--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
+  `/health` reports `queue_cap 32`; a request beyond the slot count queues rather than being refused,
+  and the 400 class is a single request over `--max-model-len`.
 - **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
   slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
   256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
@@ -280,41 +303,104 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
   in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
   GiB, refused).
-- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
+- **Decode pays for the chunk it shares a step with, and `--busy-prefill-chunk auto` does not
-  3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
+  protect it.** Re-measured on the live 1.8.1 shape (34.8 h boot, Prometheus at 60 s,
-  1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
+  `engine_step_seconds{,_decode}_total` / `engine_steps{,_decode,_mixed,_prefill}_total` /
-  at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
+  `prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a
-  chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
+  chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's
-  decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
+  step time**, so a session decoding while another prompts advances one token per 166 ms. Bucket the
-  samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
+  harness's `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` (15 s
-  `prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
+  scrapes; per-stream rate = the gauge ÷ `vllm:num_requests_running`): median **166 t/s** under light
-  bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
+  prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×** collapse, and at 13 t/s with two
-  0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
+  sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing
-  busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
+  else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute,
-  confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
+  17.9 ms in the deep one) and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from
-  resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
+  30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second
-  [[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
+  victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s**
-  and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
+  and 15.5 % of requests wait over 10 s for their first token, with `http_queue_depth` at 2-3, while
-- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
+  median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264
-  shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
+  `vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The
-  the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
+  chunk is `min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here,
-  step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
+  and `busy_cap` is the fourth law.
-  the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
+- **`auto` is a prefill-speed dial, not a decoder-protection dial.** `Scheduler::busy_cap` reserves
-  The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
+  `1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the
-  knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
+  budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders:
-  edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
+  "not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and
-  it is trusted.
+  the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while
-- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
+  anything decodes behind it, which costs more" (`core/sched/scheduler.cpp:874-878`, 1.8.1). Live
-  adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
+  proof: `prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes
-  declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
+  with a waiter, not the minutes with a decoder. An explicit `N` is the only value that applies
-  (single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
+  "whenever the chunk shares the step".
-  today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
+- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a
-  floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
+  ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count —
-  moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
+  `max(n_tok, n_seq × (1 + spec))`, `core/engine.cpp:1142` — so a 1024-token chunk with two decode
-  the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
+  rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the
-  mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
+  prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on
-  retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
+  the boot line `prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48
-  fp8-native start.
+  routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its
+  pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The
+  vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt
+  beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**.
+- **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena
+  lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims
+  it **synchronously before the step** — units revert to their host copies (an address change, not a
+  copy) and both read streams drain — and `arena_tick` regrows the loan only **2 s after the last big
+  step** (`core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs
+  with a slab ~780 expert units smaller over a link also carrying the staged copies. `/diagnostics`
+  says the binding constraint in one line: "a prefill step streams every routed layer's pooled
+  experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
+  Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
+- **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
+  N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
+  costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
+  two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
+  ~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
+  measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
+  substitute for it.
+- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N
+  intervals, so `--prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2
+  intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span
+  starts to win), while **every other step is planned at the shared budget of 2048** — which is a
+  lending level *and* keeps the stager's ring, the one combination `--busy-prefill-chunk` cannot
+  express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's
+  cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a
+  span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's
+  second box (2×R9700, w4nl64, TP2) it was TTFT-neutral — 2.61 vs 2.56 s at 18k, 10.72 vs 10.55 at
+  75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096
+  does not boot at 3 slots; this shape is 2.
+- **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over
+  the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 %
+  replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last
+  **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`,
+  `core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about
+  `log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as
+  deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840
+  · … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every
+  `prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the
+  cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly
+  **`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs
+  retention only - the writes happen at every interval whatever the policy says (only `--prefill-span`
+  skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM
+  alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB
+  against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` -
+  the pinning room left beside the expert pool at boot - as the hard bound the ledger line names.
+  Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a
+  chain that keeps resuming at one deep point stops paying for it. Shortening
+  `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
+  `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
+  interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
+- **Slots cost KV, host RAM, rows, and pin — never speed.** `--max-num-seqs` carries no radiance
+  constraint: it adds ≈110 MiB of recurrent state per slot, 10 × 111.98 MiB of pinned checkpoint-store
+  RAM per slot (`--checkpoint-slots auto`), and declares the fused TP all-reduce over
+  `seqs × (1 + spec)` rows — `wht6` is bit-exact up to 25 rows, and the live shape is 4 × 3 = **12**, so
+  single-sequence decode stays exact and so does any step up to that many rows. Raising it also raises
+  the cache's *demand* (`ceiling = slots × window`), which the resolver funds from `elastic −
+  expert_floor` — so a slot count without matching host room is paid for by expert residency. Measured
+  2026-10-06: seqs 2 and 3 statistically tied (30.9-33.1 t/s, inside the ±3 % floor). Measured
+  2026-10-19 → 10-20 at 1.8.1: 2 → 4 slots cost **+4,224 MiB of pinned host pool** and **−3.01 GiB of
+  expert residency a card** and bought 2.76 → 5.64 GiB of cache (365,204 → 747,492 tokens). The old
+  "seqs 2 cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
+  specialization, and radiance booted seqs 2 on its first fp8-native start.
 - **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
   of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
   pinned weights, and the doubling is invisible in the weight count. Read the engine log for
@@ -396,12 +482,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
   theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
   60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
-  software theory.
+  software theory. **That mitigation is a BIOS setting and it is not in effect now:** `sudo
+  dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event
+  between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed
+  before crediting any software revert with a fix — and before believing a clean run of the arm
+  you just tested, because the confounder is still in the slot.
 - **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
   `Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
   all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
   them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
   decode, `exact` is the bit-exact alternative.
+- **The link is already at the board's ceiling — do not chase the `link-below-max` warning.**
+  Both root ports (`00:01.1`, `00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is
+  the whole budget per card and the cards' own `pcie_dpm` tables expose only x8 states.
+  `/diagnostics` still warns that the on-card switch hop (`01:00.0`, `05:00.0`) runs "x8 of its
+  x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host
+  side of the chain is x8 max, which the same endpoint reports two findings later as
+  `link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so
+  the only lever on a link-bound prefill is **less traffic over the link** (more expert
+  residency, fewer pooled experts), never a wider link. P2P resolves **on** (`auto` asks the
+  driver, the two cards reach each other's memory), so the collectives do not hairpin through host
+  RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the
+  all-reduce wins the link, the staged experts wait.
+- **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8
+  cores / 8 threads, one NUMA node, `nodes: -1` for both cards) running Jellyfin, Plex, Calibre
+  and RomM beside the engine, and `--http-threads` defaults to one thread per core. Measured
+  during a heavy prefill: load average ≈ 1.0, `jellyfin` 11 % CPU, `radiance` 7 % — **CPU
+  contention is not the current decode culprit**, so do not spend an experiment window on it;
+  re-read `uptime` and `ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks
+  different from the step-sharing one.
 - `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
   the disk tier silently never opens. The tier does survive a payload swap when arch and
   tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
@@ -445,11 +554,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 ## Contract (the wire)
 
 - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
-  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
+  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`, **`/diagnostics`**. `/health`
-  JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
+  answers 200 JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while
-  (`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
+  stopping** (`core/server/admin.cpp:385`), and the listener binds *after* the full load
-  `engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
+  (`main.cpp`: `engine.start()` then `engine.serve()`) — during the load the port refuses
-  bare `curl -fsS …/health` gate is honest and needs no body grep.
+  connections, so a bare `curl -fsS …/health` gate is honest and needs no body grep.
+- **`/diagnostics` is the instrument for "is the box, not the config, the limit"** — one call
+  returns `configuration` (every flag, so drift from the env file is checkable without the host),
+  `environment`, `derived` (the VRAM split as the ledger prints it), `hardware` (host, per-card
+  `link_chain` + `pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and `findings`
+  (level / what / why / fix). On this box it reports `experts-from-host` (5.99 GiB a rank crosses
+  PCIe when used), `link-capped` (root ports are x8 max — the board), `swap-in-use`,
+  `tp-wire-lossy`, and **`link-below-max` as a `warn` at 100 % card busy on both cards** — which is
+  structural on this board; see the x8-ceiling law under *This host, and the deploy path*.
 - **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
   only the alias; container, unit, scripts and env keep the service name). The env's contract
   line is that **radiance validates the requested model name, so every client — harness, bench
@@ -476,12 +593,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
   — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
 - **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
-  longer than the KV pool's token count is accepted then refused as too large to fit; under the
+  longer than the KV pool's token count is accepted then refused as too large to fit; a request
-  3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
+  beyond `--max-num-seqs` is held by the HTTP port (`queue_cap` = `max(8, 8 × slots)` — **32** here)
+  and answered 503 only past that, never paged or squeezed. The cache funds **3.7** full-height
+  sessions, so four conversations can be resident but a fourth near-full-height one can still
+  **preempt a running sequence from the back** to free its blocks
+  (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so its session halts and then
+  re-prefills. Meter: `vllm:num_preemptions_total` (cumulative for the boot, `/stats preemptions`);
+  `radiance:num_requests_preempted` is only the instant depth.
 - **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
-  Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
+  Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the
-  published-port quirk is same-host only). The boot ledger — the referee for every shape number
+  netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under
-  above — is in the container log (`podman logs frank-archon`)
+  `radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the
+  running/waiting gauges are `vllm:*` with no radiance-native equivalent, so a `radiance:`-only
+  query misses every latency answer. The step-phase counters are all scraped:
+  `engine_steps{,_decode,_mixed,_prefill,_held}_total`, `engine_step_seconds{,_decode}_total`,
+  `engine_prefill_chunks_capped_total`, `prefill_tokens_total`, `decode_tokens_per_second`,
+  `prefill_tokens_per_second`, `prefix_hit_cut_tokens_total`, `http_queue_depth`. Two holes: the
+  engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
+  `token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine /
+  prefill stager have no metric at all** — that half of every law above is read from the boot ledger,
+  `/stats` and `/diagnostics`, never from a graph. The boot ledger — the referee for every shape
+  number here — is in the container log (`podman logs frank-archon`);
   `/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
 
 ## Files

serverAppConfigs-docs/LLM/frank-dragoon.md

@@ -10,14 +10,15 @@ tags:
   - gfx1151
   - halogen
   - endpoint
-timestamp: 2026-10-19
+timestamp: 2026-10-21
 ---
 
 # frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
 
 Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
 under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
-(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
+(`2b12716`), ceiling raised to the native 262,144 on 2026-10-21 so the harness window is 200,000.
+This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
 
 ## Current shape
 
@@ -37,10 +38,12 @@ Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b
 unauthenticated token port 8730 stays on container loopback. Engine stdout →
 `/home/dank/logs/frank_dragoon.log`, **not** the journal.
 
-- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
+- Shape: `HALOGEN_CTX=262144` hard per-request ceiling = the checkpoint's native window (raised
-  `HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
+  2026-10-21 from 200704; `HALOGEN_ROPE_YARN` beyond native is not on plan) · `HALOGEN_KV_SLOTS=2` ·
-  enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
+  `HALOGEN_KV_POOL_POSITIONS=401408` inside the band `tests/frank_dragoon_unit.bats` enforces
-  exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
+  (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
+  exists here, the pool is the only KV lever · the pool carries TWO maximum harness-length (200k)
+  conversations at once; two literal ceiling-length requests would queue — that is a pool raise.
 - Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
   omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
   pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
@@ -53,10 +56,11 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
   (Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
   ~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
   Caddy/observability).
-- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
+- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 200000`
-  (= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
+  — sits 62,144 under the ceiling, clearing the ~45k fit-law gap (400-zone law below);
-  `reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
+  `tests/frank_dragoon_unit.bats` fails if the page's declared window and `HALOGEN_CTX` drift back
-  rides [[frank-zealot]].
+  together. `maxTokens: 45056`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`,
+  `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction rides [[frank-zealot]].
 - Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
   pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
   Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
@@ -124,14 +128,18 @@ created it — earlier topologies in .
   prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
   85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
   re-measure the warm ratio after any pool change.
-- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
+- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling, because the harness
-  155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
+  clamps `max_tokens` to `contextWindow − est − 4096`, its chars/4 `est` UNDER-counts code/agent
-  its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
+  traffic by ~3 %, and the compaction trigger fires at 95 % of the window (`config.yml
-  3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
+  thresholdPercent`, while the comments designed around 85 % — check the trigger BEFORE touching a
-  over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
+  window; that disagreement is why long sessions kept drawing 400s). The 3 % is measured, not
-  target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
+  theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 — over by 354, a 400.
-  is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
+  At the live pair (ceiling 262,144 / window 200,000) the worst-case ordinary request is
-  under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
+  200,000 − 4,096 + 3 % ≈ 201.8k, and the trigger point is 190,000 est × 1.03 + 45,056 ≈ 240.7k —
+  both inside the ceiling with ~60k of margin. Compaction rides [[frank-zealot]] at
+  ≈ threshold − keepRecent(20k) ≈ 175k measured against its 200,704-token slot. The seam in
+  `tests/frank_dragoon_unit.bats` re-runs this arithmetic at the 95 % trigger and reads the window
+  from THIS page, so ceiling/window pairs that drift back together fail the suite, not a session.
 - **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
   503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
   together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
@@ -248,9 +256,14 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
    (`dankclaw/udev/98-halogen-npu-access.rules`, installed and re-triggered by the deployer, `0666` like the GPU
    compute nodes beside it); the serve script refuses any node that is not other-accessible, so the fault cannot
    return as a crash-loop.
-2. The host's XRT reachable inside: there is **no `/opt/xilinx/xrt`** here — distro packages put the three files the
+2. The host's XRT reachable inside, at **THREE prefixes per library**: there is **no `/opt/xilinx/xrt`** here —
-   container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`) in
+   distro packages put the three files the container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`,
-   `/usr/lib/x86_64-linux-gnu`, where all three exist.
+   `libxrt_driver_xdna.so.2`) in `/usr/lib/x86_64-linux-gnu`. The serve script mounts each REAL file (never a
+   symlink) at `/opt/xilinx/xrt/lib/`, at its own path, AND at `/opt/xilinx/lib/<multiarch>/` — the NPU child
+   searches that third prefix and a missing one is fatal: since the host's `libxrt2` was reconfigured
+   2026-10-20 the child died with `No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the
+   engine supervisor took the WHOLE container down on it (88-restart loop, chat dead beside a healthy GPU line).
+   A dead lane is not survivable here — arm the lane only with all three prefixes mounted.
 3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
    while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
    engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**

shared/alloy/config.alloy

@@ -1,35 +1,68 @@
 // Alloy configuration — ships host and container logs to Loki.
 // One instance per host. Deployed to /opt/alloy/config/config.alloy.
+//
+// Runs as a ROOT system service on Network=host (see
+// dankclaw/observability/quadlet/alloy.container), so it reaches Loki on the
+// host's loopback and reads every journal — host services and rootless-quadlet
+// containers alike.
 
 logging {
   level = "info"
 }
 
-// The alloy UI service listens on 127.0.0.1:12345 by default; the
+// The alloy UI listens on 127.0.0.1:12345; the in-container healthcheck probes
-// in-container healthcheck probes it there. alloy 1.x refuses to
+// it there. alloy 1.x refuses to configure the ui service from a config block,
-// configure the ui service from a config block, so there is no block.
+// so there is no ui block.
 
-// Ship logs to Loki (container name on the podman network).
+// Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
+// host network, so it pushes by loopback, never by container name — a root
+// container is not on the rootless dankclaw_default network.
+// GRAFANA_HOST_LABEL is replaced with this host's name by
+// copy_alloy_config at deploy time; it must not appear verbatim in the
+// installed config.
 loki.write "default" {
   endpoint {
-    url = "http://loki:3100/loki/api/v1/push"
+    url = "http://127.0.0.1:3100/loki/api/v1/push"
   }
 }
 
-// Systemd journal from the host socket.
+// The systemd journal. As root with /etc/machine-id + the journal dirs mounted,
+// this one source reads system.journal AND user-*.journal, so it carries both
+// host services and the rootless Quadlet containers' logs. There is deliberately
+// no container-file source: podman container stdout lands in the journal, and
+// the old /var/lib/containers/.../ctr.log and /var/log/containers globs pointed
+// at a root-storage layout that does not exist on this host.
 loki.source.journal "journal" {
   format_as_json = true
   labels = {
-    job = "systemd-journal",
+    job  = "systemd-journal",
+    host = "GRAFANA_HOST_LABEL",
+  }
+  forward_to = [loki.process.relabel.receiver]
+}
+
+// Promote the fields that ARE the point of the log stream to real Loki labels.
+// Without this the journal arrives as one opaque JSON line per entry and every
+// per-container view is structurally impossible: Loki metrics queries can only
+// group by INDEXED stream labels, so `sum by (container_name)` returns a single
+// anonymous series and the logs-overview board read empty beside 6.5k
+// lines/5min of live traffic (measured 2026-10-21). CONTAINER_NAME is what
+// podman's journald logging writes (uppercase); host units lack it and stay on
+// the host-only stream. `format_as_json = true` above guarantees every line
+// reaching this stage is a JSON object, so there is no non-JSON branch to
+// protect — alloy 1.21's json stage has no `on_error` attribute (validated
+// against the pinned image); a malformed line would drop, which is the
+// correct fate for a line Loki could not index anyway.
+loki.process "relabel" {
+  stage.json {
+    expressions = {
+      container = "CONTAINER_NAME",
+    }
+  }
+  stage.labels {
+    values = {
+      container = "container",
+    }
   }
   forward_to = [loki.write.default.receiver]
 }
-
-// Container logs (podman storage path; adjust for docker).
-loki.source.file "containers" {
-  targets = [
-    {__path__ = "/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log", job = "podman"},
-    {__path__ = "/var/log/containers/*.log", job = "containers"},
-  ]
-  forward_to = [loki.write.default.receiver]
-}

shared/blackbox/blackbox.yml

@@ -0,0 +1,17 @@
+# Blackbox exporter modules — the honest answer to "is this HTTP service
+# answering?". A Prometheus scrape cannot express it (an HTML front page is a
+# 200 that fails the scrape protocol), which is why the old `services` and
+# `comfyui` jobs sat permanently red on `unsupported Content-Type`.
+# See serverAppConfigs-docs/Architecture/Stale Signal Audit.md §1.
+# The listen address is NOT set here: the quadlet unit's Exec passes
+# --listen-address (LAN bind — see the unit header for why loopback is
+# unreachable from the bridge-networked Prometheus). Keep this file to
+# modules only, so the two sources cannot disagree.
+
+modules:
+  http_2xx:
+    prober: http
+    timeout: 10s
+    http:
+      preferred_ip_protocol: ip4
+      valid_status_codes: [] # default: 2xx + 3xx

shared/grafana/provisioning/dashboards/containers.json

@@ -0,0 +1,137 @@
+{
+  "uid": "containers",
+  "title": "Containers — Dankclaw & Dankserver",
+  "description": "Per-container CPU and memory for the quadlet app stacks, from cadvisor (a rootless quadlet on each host). cadvisor keys each series by the cgroup path, and Prometheus relabels the quadlet unit name (app.slice/<name>.service) into a `container` label. Panels use only the metric families cadvisor actually exports for these rootless containers: CPU usage and memory working-set/limit/OOM. Network and filesystem are NOT exported for them (no netns/classid attribution under the rootless+host-PID layout), so no panels pretend otherwise. See serverAppConfigs-docs/Infrastructure/Observability.md.",
+  "tags": ["fleet", "cadvisor", "containers"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "host",
+        "label": "Host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\"}, host)", "refId": "host_var" },
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      },
+      {
+        "name": "container",
+        "label": "Container",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\"}, container)", "refId": "container_var" },
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "row",
+      "title": "CPU",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+      "panels": []
+    },
+    {
+      "id": 2,
+      "type": "timeseries",
+      "title": "CPU usage (cores)",
+      "description": "CPU-seconds/s per container = cores busy. A steady high line is a busy service; a spike is a burst (a render, an index, a backup). Only containers are shown, not the host or its daemons.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 1 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "sum by (host, container) (rate(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m]))",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "bargauge",
+      "title": "Top containers by CPU (cores)",
+      "description": "Instant ranking over the selected window. Tells you which container to look at when the host is loaded.",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 1 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "max": null, "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 4 } ] } }, "overrides": [] },
+      "options": { "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false }, "displayMode": "gradient", "orientation": "horizontal", "showUnfilled": true },
+      "targets": [
+        {
+          "expr": "topk(12, sum by (host, container) (rate(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m])))",
+          "legendFormat": "{{host}} / {{container}}",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "row",
+      "title": "Memory",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 10 },
+      "panels": []
+    },
+    {
+      "id": 5,
+      "type": "timeseries",
+      "title": "Memory working set",
+      "description": "container_memory_working_set_bytes per container — the number the OOM killer acts on. Cache-inclusive and reclaimable; a flat high line is resident data, a sawtooth is churn.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 11 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "timeseries",
+      "title": "Memory vs host total (%)",
+      "description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "type": "timeseries",
+      "title": "OOM kills",
+      "description": "container_oom_events_total rate. Any non-zero line means the kernel killed a process inside that container — the honest signal that its memory limit is too tight for its real working set.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 20, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "sum by (host, container) (rate(container_oom_events_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m]))",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/host-system.json

@@ -0,0 +1,204 @@
+{
+  "uid": "host-system",
+  "title": "Hosts \u2014 Dankclaw & Dankserver",
+  "description": "node_exporter for both servers. Starter set \u2014 throw-at-the-wall breadth; refine the windows and drop the noise. The picker selects the host; CPU/load/memory/filesystem/network are per-node.",
+  "tags": ["host", "system"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "node",
+        "label": "host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(up{job=~\"node-.*\"}, job)", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      },
+      {
+        "name": "gpuhost",
+        "label": "gpu host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(gpu_utilization_percent, host)", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1, "type": "stat", "title": "Uptime",
+      "description": "Time since boot. A drop means a reboot.",
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [{ "expr": "time() - node_boot_time_seconds{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "stat", "title": "Memory used",
+      "description": "1 - MemAvailable/MemTotal. The honest \"in use\" figure (excludes reclaimable cache).",
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.8 }, { "color": "red", "value": 0.92 }] } }, "overrides": [] },
+      "targets": [{ "expr": "1 - node_memory_MemAvailable_bytes{job=~\"$node\"} / node_memory_MemTotal_bytes{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "stat", "title": "Load 5m / core",
+      "description": "Per-core load5. >1 sustained means runnable tasks are queuing behind cores.",
+      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "stat", "title": "Open file handles",
+      "description": "FDs allocated vs the kernel maximum. Rising toward the max precedes EMFILE across the host.",
+      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 0.9 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_filefd_allocated{job=~\"$node\"} / node_filefd_maximum{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "timeseries", "title": "CPU busy %",
+      "description": "CPU seconds/s in each mode = cores busy per mode; the stack sums to total cores in use (16/host here). A user or system spike points at what is loading the box; steal > 0 means the hypervisor is throttling.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 10, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,mode) (rate(node_cpu_seconds_total{job=~\"$node\",mode!=\"idle\",mode!=\"iowait\"}[5m]))", "legendFormat": "{{job}} {{mode}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 6, "type": "timeseries", "title": "Load average",
+      "description": "load1/5/15 per host.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_load1{job=~\"$node\"}", "legendFormat": "{{job}} load1", "refId": "A" },
+        { "expr": "node_load5{job=~\"$node\"}", "legendFormat": "{{job}} load5", "refId": "B" },
+        { "expr": "node_load15{job=~\"$node\"}", "legendFormat": "{{job}} load15", "refId": "C" }
+      ]
+    },
+    {
+      "id": 7, "type": "timeseries", "title": "Memory breakdown",
+      "description": "Total, free, buffers+cache. Reclaimable cache is not pressure; MemAvailable is the real headroom.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_memory_MemTotal_bytes{job=~\"$node\"} - node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} used", "refId": "A" },
+        { "expr": "node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} available", "refId": "B" }
+      ]
+    },
+    {
+      "id": 8, "type": "timeseries", "title": "Filesystem free %",
+      "description": "Per mountpoint. The media/library stacks fill disks.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 0.15 }] } }, "overrides": [] },
+      "targets": [
+        { "expr": "node_filesystem_avail_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"} / node_filesystem_size_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"}", "legendFormat": "{{job}} {{mountpoint}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 9, "type": "timeseries", "title": "Disk throughput",
+      "description": "Read/write bytes/s per device.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_disk_read_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} R", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_disk_written_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} W", "refId": "B" }
+      ]
+    },
+    {
+      "id": 10, "type": "timeseries", "title": "Disk busy %",
+      "description": "Fraction of time the device had I/O in flight. Near 1 = the device is the bottleneck.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] },
+      "targets": [
+        { "expr": "rate(node_disk_io_time_seconds_total{job=~\"$node\"}[5m])", "legendFormat": "{{job}} {{device}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 11, "type": "timeseries", "title": "Network throughput",
+      "description": "Receive (up) / transmit (down) bytes/s per interface.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} RX", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_network_transmit_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} TX", "refId": "B" }
+      ]
+    },
+    {
+      "id": 12, "type": "timeseries", "title": "Network errors / drops",
+      "description": "Interface errors and drops \u2014 the signature of a saturated link or a bad NIC/cable.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} RX err", "refId": "A" },
+        { "expr": "sum by (job,device) (rate(node_network_transmit_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} TX err", "refId": "B" }
+      ]
+    },
+    {
+      "id": 13, "type": "timeseries", "title": "GPU busy %",
+      "description": "Per-GPU utilization from gpu-exporter (amdgpu gpu_busy_percent / nvidia-smi). The picker above selects the host; each series is one card.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 36 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_utilization_percent{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 14, "type": "timeseries", "title": "GPU VRAM used",
+      "description": "Device memory in use. On the R9700s (frank-archon) and the 3090 (frank-zealot) this is the loaded model's footprint.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 36 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_memory_used_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 15, "type": "timeseries", "title": "GPU VRAM used %",
+      "description": "Used / total device memory. Near 100% is the pre-OOM wall.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 44 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
+      "targets": [
+        { "expr": "100 * gpu_memory_used_bytes{host=~\"$gpuhost\"} / gpu_memory_total_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 16, "type": "timeseries", "title": "GPU temperature",
+      "description": "Package temperature. AMD edge temp (hwmon temp1_input); NVIDIA gpu temp.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 44 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "celsius", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_temperature_celsius{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 17, "type": "timeseries", "title": "GPU power draw",
+      "description": "Board power draw. Emitted only where readable: NVIDIA and the Dankclaw iGPU. The R9700s have no sysfs power sensor, so they are simply absent here — never a fabricated 0.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 52 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "watt", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_power_watts{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/llm-fleet.json

@@ -194,7 +194,7 @@
       "id": 6,
       "type": "timeseries",
       "title": "Firing fleet alerts",
-      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
+      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
       "gridPos": {
         "h": 8,
         "w": 12,

shared/grafana/provisioning/dashboards/logs-overview.json

@@ -0,0 +1,243 @@
+{
+  "uid": "logs-overview",
+  "title": "Logs \u2014 Loki (systemd journal)",
+  "description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
+  "tags": [
+    "logs",
+    "loki"
+  ],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "templating": {
+    "list": [
+      {
+        "name": "container",
+        "label": "container",
+        "type": "query",
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "query": {
+          "query": "label_values({job=\"systemd-journal\"}, container)",
+          "refId": "LokiVariableQueryEditor-VariableQuery"
+        },
+        "includeAll": true,
+        "multi": true,
+        "refresh": 2,
+        "current": {
+          "text": "All",
+          "value": "$__all"
+        }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "timeseries",
+      "title": "Log volume by container (lines/s)",
+      "description": "Per-container journal line rate. The `container` label is promoted at ingestion by Alloy (loki.process) from podman's CONTAINER_NAME journald field - Loki metrics can only group by indexed stream labels.",
+      "gridPos": {
+        "h": 9,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "custom": {
+            "fillOpacity": 20,
+            "stacking": {
+              "mode": "normal"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 2,
+      "type": "timeseries",
+      "title": "Total log rate",
+      "description": "All journal lines per second.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))",
+          "legendFormat": "all lines/s",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "timeseries",
+      "title": "Error-ish line rate (text match)",
+      "description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "color": {
+            "mode": "fixed",
+            "fixedColor": "red"
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))",
+          "legendFormat": "error lines/s",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "timeseries",
+      "title": "Errors by container",
+      "description": "Which unit is throwing. Rank by the top offenders.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} |~ \"(?i)(error|panic|fatal)\" [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 5,
+      "type": "stat",
+      "title": "Error lines (last 15m)",
+      "description": "Absolute count over the window, not a rate.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 50
+              },
+              {
+                "color": "red",
+                "value": 500
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))",
+          "instant": true,
+          "legendFormat": "errors/15m",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "logs",
+      "title": "Journal (filtered)",
+      "description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
+      "gridPos": {
+        "h": 12,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "options": {
+        "showTime": true,
+        "wrapLogMessage": true,
+        "sortOrder": "Descending",
+        "enableLogDetails": true
+      },
+      "targets": [
+        {
+          "expr": "{job=\"systemd-journal\", container=~\"$container\"}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/observability.json

@@ -0,0 +1,385 @@
+{
+  "uid": "observability",
+  "title": "Observability \u2014 self-monitoring",
+  "description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
+  "tags": [
+    "observability",
+    "meta"
+  ],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "stat",
+      "title": "Targets up",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 1)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 2,
+      "type": "stat",
+      "title": "Targets down",
+      "description": "Every scrape target that is down. ZERO IS THE HEALTHY STATE: the HTML-probe jobs and the disabled Gitea metrics were retired 2026-10-21 (HTTP liveness now rides the blackbox exporter, job \"services\"). A non-zero count here is a real regression - read /api/v1/targets lastError before triaging.",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 4,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 0)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "stat",
+      "title": "TSDB head series",
+      "description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 8,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_tsdb_head_series",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "stat",
+      "title": "Discovered targets",
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 13,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(prometheus_sd_discovered_targets)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 5,
+      "type": "stat",
+      "title": "Prometheus mem",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "process_resident_memory_bytes{job=\"prometheus\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "stat",
+      "title": "Grafana up",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            {
+              "type": "value",
+              "options": {
+                "1": {
+                  "text": "UP",
+                  "color": "green"
+                },
+                "0": {
+                  "text": "DOWN",
+                  "color": "red"
+                }
+              }
+            }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "up{job=\"grafana\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "type": "table",
+      "title": "Target health",
+      "description": "Every scrape target and its current up. Sort by Value to surface regressions.",
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "options": {
+        "sortBy": [
+          {
+            "displayName": "Value",
+            "sortOrder": "Asc"
+          }
+        ]
+      },
+      "targets": [
+        {
+          "expr": "up",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 8,
+      "type": "timeseries",
+      "title": "Scrape interval drift (p99 actual)",
+      "description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 9,
+      "type": "timeseries",
+      "title": "Scrape samples/s",
+      "description": "Total samples ingested across all jobs.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(rate(scrape_samples_scraped[5m]))",
+          "legendFormat": "samples/s",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 10,
+      "type": "timeseries",
+      "title": "Loki request rate",
+      "description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))",
+          "legendFormat": "{{route}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 11,
+      "type": "timeseries",
+      "title": "Exporter process CPU",
+      "description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 21
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))",
+          "legendFormat": "{{job}}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/datasources/loki.yml

@@ -0,0 +1,17 @@
+# Grafana datasource for Loki, declared in git rather than clicked into
+# grafana.db — same reason as the Prometheus datasource: a UI-configured
+# datasource is invisible to review, lost with the data volume, and unknown
+# to the repository. Loki OSS ships no web UI of its own, so Explore through
+# this datasource is the only way a human reads the log store.
+#
+# URL is the container name on dankclaw_default (the same address Caddy
+# proxies to), not the published loopback port.
+apiVersion: 1
+
+datasources:
+  - name: Loki
+    uid: loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    editable: false

shared/prometheus/prometheus.yml

@@ -60,20 +60,33 @@ scrape_configs:
       - targets: ["loki:3100"]
     metrics_path: "/metrics"
 
-  # ComfyUI (no native /metrics; probe the UI)
+  # HTTP liveness, probed honestly through the blackbox exporter. Scraping a
-  - job_name: "comfyui"
+  # web root is NOT a health check — an HTML front page fails the scrape
-    metrics_path: "/"
+  # protocol (`unsupported Content-Type`), which is exactly why the old
-    static_configs:
+  # `services` and `comfyui` jobs were red forever beside healthy services
-      - targets: ["comfyui:8188"]
+  # (Stale Signal Audit §1). The prober is a host-netns container listening on
-
+  # the podman gateway (the scrape address below - the netavark hairpin gap
-  # Blackbox-style probes for service up/down (published LAN ports;
+  # bars this container from its own host's LAN IP, cadvisor job note) and on
-  # gitea and comfyui are covered by their own jobs above)
+  # the LAN; from that netns it reaches the host LAN IP, and published ports
+  # via 127.0.0.1 - bridge container names do NOT resolve in the host netns,
+  # which is why comfyui's probe is a loopback URL. comfyui's own
+  # metrics-scrape job is retired, one owner per service.
   - job_name: "services"
-    metrics_path: "/"
+    metrics_path: /probe
+    params:
+      module: [http_2xx]
     static_configs:
       - targets:
-          - "192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
+          - "http://192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
-          - "192.168.0.143:3005"   # OpenWebUI
+          - "http://192.168.0.143:3005"   # OpenWebUI
+          - "http://127.0.0.1:8188"        # ComfyUI UI (published 0.0.0.0:8188; probed from the prober's host netns)
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: "host.containers.internal:9115"
 
   # LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
   # this container, not the guarded vhosts: these are machine clients and
@@ -131,3 +144,88 @@ scrape_configs:
       - targets: ["192.168.0.235:7770"]
         labels:
           service: frank-archon
+
+  # Per-container metrics from cadvisor (a rootless quadlet on each host).
+  # cadvisor keys every series by the cgroup path; for a quadlet container that
+  # path is …/app.slice/<name>.service[/libpod-payload-<hex>], so the unit NAME
+  # lives in it. The relabel below pulls that name out into a `container` label
+  # and keeps only the per-unit aggregate — cadvisor's raw host root,
+  # system.slice daemons and machine.slice scopes carry no such name and are
+  # dropped, as is the per-CPU-core split (only cpu="total" survives), so the
+  # scrape stays small. cadvisor cannot bind per-interface, so it is reached at
+  # the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
+  # DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
+  # connection whether the port is a host-net bind or a published port (the
+  # original note here claimed published ports were exempt; measured 2026-10-21
+  # from inside this container: LAN refused, gateway answers). Cross-host
+  # targets (dankserver) have no hairpin problem and stay plain LAN.
+  - job_name: "cadvisor-dankclaw"
+    static_configs:
+      - targets: ["host.containers.internal:8080"]
+        labels:
+          host: dankclaw
+    metric_relabel_configs:
+      - source_labels: [id]
+        regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
+        target_label: container
+        replacement: '$1'
+      # Keep only the unit-level cgroup; drop its payload/control children so a
+      # container is not counted twice.
+      - source_labels: [id]
+        regex: '.*/\.service/.*'
+        action: drop
+      # Drop everything that is not a named unit (no `container` was extracted).
+      - source_labels: [container]
+        regex: '.+'
+        action: keep
+      # cpu="total" for cpu series; metrics with no cpu label (memory, network,
+      # filesystem) are kept.
+      - source_labels: [cpu]
+        regex: '^$|total'
+        action: keep
+      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
+        action: labeldrop
+
+  - job_name: "cadvisor-dankserver"
+    static_configs:
+      - targets: ["192.168.0.235:8080"]
+        labels:
+          host: dankserver
+    metric_relabel_configs:
+      - source_labels: [id]
+        regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
+        target_label: container
+        replacement: '$1'
+      - source_labels: [id]
+        regex: '.*/\.service/.*'
+        action: drop
+      - source_labels: [container]
+        regex: '.+'
+        action: keep
+      - source_labels: [cpu]
+        regex: '^$|total'
+        action: keep
+      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
+        action: labeldrop
+
+  # Per-GPU metrics (gpu_utilization_percent / gpu_memory_used_bytes /
+  # gpu_memory_total_bytes / gpu_temperature_celsius / gpu_power_watts) from the
+  # host-process gpu-exporter (scripts/gpu-exporter.py). One job per host, same
+  # reach rules as node/cadvisor: Dankclaw is scraped at the podman gateway
+  # because the netavark hairpin quirk makes Prometheus fail to dial its own
+  # host LAN IP; Dankserver is plain LAN. The gpu label (PCI BDF on AMD, index on
+  # NVIDIA) + vendor/model come from the exporter, so the GPU row on the
+  # host-system dashboard filters by host. Power is emitted only where readable
+  # (NVIDIA + the Dankclaw iGPU); the R9700s have no sysfs power, so they simply
+  # have no gpu_power_watts series — no fabricated zero.
+  - job_name: "gpu-dankclaw"
+    static_configs:
+      - targets: ["host.containers.internal:9401"]
+        labels:
+          host: dankclaw
+
+  - job_name: "gpu-dankserver"
+    static_configs:
+      - targets: ["192.168.0.235:9401"]
+        labels:
+          host: dankserver

shared/prometheus/rules/llm-fleet.yml

@@ -1,10 +1,15 @@
 # Alerting and recording rules for the LLM fleet.
 #
-# One metric namespace: llamacpp:* — frank-zealot and frank-dragoon, both
+# Two metric namespaces, because two runtime families are scraped:
-# llama.cpp with /metrics (the family verified from /metrics on 2026-09-16
+#
-# and 2026-09-20). frank-archon LEFT the family 2026-10-05: FreeToken (the
+#   * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen, which
-# PR587 ROCm engine) has no /metrics endpoint, so its scrape job was
+#     re-exports the llama.cpp metric names). Verified from /metrics on
-# deleted rather than left never-green.
+#     2026-09-16 / 2026-09-20.
+#   * vllm:* — frank-archon, the radiance engine on Dankserver since
+#     2026-10-05. It exports vLLM-compatible names, a DIFFERENT namespace, so
+#     it is a separate scrape job in prometheus.yml and the rules below are a
+#     separate group. (The old claim that its scrape job was deleted is false:
+#     it re-joined the day it left the FreeToken engine and is `up` today.)
 # (The netavark published-port quirk that forces the Dankclaw endpoints
 # onto the gateway is same-host only.)
 #
@@ -22,7 +27,8 @@
 #     cached by cached + prompt.
 #   * Speculative decoding: drafts and accepted tokens are both counters,
 #     so LlmSpecDecodeOff watches the pair.
-#   * No latency histograms exist, so there is no p95 alert.
+#   * No latency histograms in llama.cpp, so no p95 alert there; radiance's
+#     vllm:* namespace DOES export one, so the vllm-fleet group alerts on it.
 #
 # IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
 # Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
@@ -112,6 +118,84 @@ groups:
           description: >-
             Prometheus cannot reach the LLM endpoint. frank-zealot and
             frank-dragoon are root units on Dankclaw: `systemctl is-active
-            frank-zealot` / `frank-dragoon`. Both scraped endpoints export
+            frank-zealot` / `frank-dragoon` (both export llamacpp:* —
-            the llamacpp:* family. (frank-archon left the scrape set
+            dragoon via halogen's llama.cpp-compatible names). frank-archon
-            2026-10-05 — FreeToken exports no /metrics.)
+            is a radiance endpoint on Dankserver (`systemctl show
+            frank-archon` there) exporting the vllm:* family; the
+            `frank-.*` selector covers all three regardless of namespace.
+
+  # frank-archon (radiance) exports the vLLM-compatible namespace, NOT
+  # llamacpp:*. Every rule here is keyed to a series the exporter actually
+  # emits (verified from /metrics 2026-10-20: num_requests_waiting,
+  # gpu_prefix_cache_hit_rate, num_preemptions_total, prompt_tokens_total,
+  # time_to_first_token_seconds). The llamacpp rules above evaluate nothing for
+  # this job and these evaluate nothing for llama.cpp/halogen — two clean
+  # namespaces, each keyed only to what its runtime exports.
+  - name: vllm-fleet
+    interval: 30s
+    rules:
+      # p95 time-to-first-token, shared with the dashboard. When the endpoint
+      # is idle every bucket rate is zero, histogram_quantile yields NaN, and
+      # neither this nor the alert below can fire on an idle engine.
+      - record: job:vllm_ttft_p95:rate5m
+        expr: |
+          histogram_quantile(0.95, sum by (job, le)
+              (rate(vllm:time_to_first_token_seconds_bucket[5m])))
+
+      # Radiance exports a real queue-depth gauge (llama.cpp only shows the
+      # wall as requests_deferred). --max-num-seqs is small, so a sustained
+      # wait is either real load or this endpoint carrying the fleet alone.
+      - alert: VllmRequestsQueueing
+        expr: vllm:num_requests_waiting > 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }}: requests waiting for 10 minutes"
+          description: >-
+            The radiance scheduler has requests queued. Either the fleet is
+            genuinely busy or this endpoint is carrying it alone.
+
+      # Radiance reports prefix reuse directly as a hit-rate gauge — no
+      # counter division needed. The token-rate floor keeps an idle engine
+      # (hit rate 0 with no traffic) from firing.
+      - alert: VllmPrefixCacheCollapse
+        expr: >-
+          vllm:gpu_prefix_cache_hit_rate < 0.20
+            and rate(vllm:prompt_tokens_total[5m]) > 0.05
+        for: 15m
+        labels:
+          severity: info
+        annotations:
+          summary: "{{ $labels.job }}: prefix hit rate below 20 % for 15 minutes"
+          description: >-
+            Nearly every request prefilling from scratch. With several large
+            contexts in flight this is the capacity wall, not a routing fault.
+
+      # Preemption is the KV-thrash signature radiance reports that llama.cpp
+      # cannot: the engine evicts a running request to admit another. Any
+      # sustained rate means the KV cache cannot hold the concurrency in flight.
+      - alert: VllmPreempting
+        expr: rate(vllm:num_preemptions_total[5m]) > 0
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }}: engine preempting requests"
+          description: >-
+            The engine is evicting running requests to admit new ones — the KV
+            cache is undersized for the concurrency in flight.
+
+      # p95 first-token latency, kept HIGH on purpose: a large-prompt prefill
+      # legitimately spends many seconds before token 1, so this fires only on
+      # a wedged scheduler or a queue backed up behind one request.
+      - alert: VllmTtftSlow
+        expr: job:vllm_ttft_p95:rate5m > 120
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }}: p95 time-to-first-token over 120 s"
+          description: >-
+            First-token latency is pathological, not an honest big prefill.
+            Likely a wedged scheduler or one request stalling the queue.

tests/caddyfile.bats

@@ -193,7 +193,18 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
 }
 
 @test "prometheus targets are host:port (no URL schemes)" {
-    ! grep -nE '^\s*-\s*"?http' "$REPO_ROOT/shared/prometheus/prometheus.yml"
+    # Plain scrape targets are host:port; a URL target is only meaningful for
+    # the blackbox prober job, whose relabel_configs copy it to __param_target
+    # and point __address__ at the prober. Anywhere else a URL target is the
+    # old HTML-probe defect wearing a new hat.
+    awk '
+        /job_name:/ { job=$0; sub(/.*"/, "", job); sub(/".*/, "", job); probe = 0 }
+        /metrics_path: \/probe/ { probe = 1 }
+        /^[[:space:]]*-[[:space:]]*"http/ && !probe {
+            print FILENAME ":" FNR ": URL target outside a prober job: " $0; bad = 1
+        }
+        END { exit bad }
+    ' "$REPO_ROOT/shared/prometheus/prometheus.yml"
 }
 
 @test "observability containers run as container root (rootless = host user)" {
@@ -240,14 +251,12 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
 }
 
 @test "alloy's UI address comes from the unit, not the config file" {
-    # alloy 1.x refuses to configure the ui service from a config block
+    # alloy 1.x refuses to configure the ui service from a config block, so the
-    # ("service ui does not support being configured"), so the unit moves the
+    # unit sets the UI address on the command line. As a Network=host root unit
-    # UI off the container's loopback with a command-line flag. Without that
+    # that address is the host loopback directly (no PublishPort) — reachable
-    # flag the port cannot be published and the service has no reachable
+    # from the host, from nothing off-host.
-    # health endpoint at all — which is why the homepage status card for
-    # alloy read "down" while it ran under compose.
     ! grep -qE '^(ui|server) \{' "$REPO_ROOT/shared/alloy/config.alloy"
     local unit="$REPO_ROOT/dankclaw/observability/quadlet/alloy.container"
-    grep -q -- '--server.http.listen-addr=0.0.0.0:12345' "$unit"
+    grep -q -- '--server.http.listen-addr=127.0.0.1:12345' "$unit"
-    grep -q '^PublishPort=127.0.0.1:12345:12345$' "$unit"
+    ! grep -q '^PublishPort=' "$unit"
 }

tests/deployment_flow.bats

@@ -125,3 +125,48 @@ load helpers
     [ ! -e "$sandbox/frank-default-sightsound" ]
     [ -e "$BATS_TEST_TMPDIR/.AGENTTRASH/$(date +%Y%m%d)-retired-deploy-state/frank-default-sightsound" ]
 }
+
+@test "the client and remote allowlists agree per target" {
+    # Two gates guard `deploy.sh --service X`: the client's valid_service and
+    # the remote runner's service_is_allowed. When only one side gained a
+    # service the other answered "is not deployed on <target>" and the only
+    # repair path was a fleet-wide drain — cadvisor and gpu-exporter both fell
+    # through this crack within a day of each other. The lists MUST agree;
+    # searxng-network is the one sanctioned exception (client-only network
+    # unit, no remote dispatch handler).
+    run env OUT="$BATS_TEST_TMPDIR" bash -c '
+        grep -oE "dank(claw|server):[a-z0-9-]+" \
+            <(sed -n "/^valid_service()/,/^}/p" "$1/scripts/deploy.sh") \
+            | grep -v "dankclaw:searxng-network" | sort -u > "$OUT/cli"
+        grep -oE "dank(claw|server):[a-z0-9-]+" \
+            <(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
+            | sort -u > "$OUT/rem"
+        diff "$OUT/cli" "$OUT/rem"
+    ' _ "$REPO_ROOT"
+    [ "$status" -eq 0 ] || { echo "deploy.sh vs deploy-remote.sh allowlist drift:"; echo "$output"; return 1; }
+    # Drift aside, the gate must actually know the host-process exporters —
+    # their first-ever deploy has to pass the client gate.
+    for pair in dankclaw:cadvisor dankclaw:gpu-exporter dankserver:cadvisor dankserver:gpu-exporter; do
+        grep -q "$pair" "$BATS_TEST_TMPDIR/cli" || { echo "client gate missing $pair"; return 1; }
+    done
+}
+
+@test "every gated dankclaw service is dispatched" {
+    # A service can clear both allowlists and still die inside the deployer:
+    # dankclaw has no compose file, and deploy_compose_service (the case
+    # catch-all) hard-fails with "no quadlet handler" since 2026-09-16. So
+    # every name the remote gate allows for dankclaw MUST be named by a
+    # deploy_service pattern — blackbox-exporter (2026-10-21) shipped with
+    # both gates and the health gate wired but no dispatch case, and only
+    # the deploy attempt found it.
+    run env OUT="$BATS_TEST_TMPDIR" bash -c '
+        grep -oE "dankclaw:[a-z0-9-]+" \
+            <(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
+            | cut -d: -f2 | sort -u > "$OUT/gated"
+        sed -n "/^deploy_service()/,/^}/p" "$1/scripts/deploy-remote.sh" \
+            | grep -oE "^ +[a-z][a-z0-9|-]*\)" | tr -d " )" | tr "|" "\n" \
+            | sort -u > "$OUT/dispatched"
+        comm -23 "$OUT/gated" "$OUT/dispatched"
+    ' _ "$REPO_ROOT"
+    [ -z "$output" ] || { echo "gated dankclaw services with no deploy_service case:"; echo "$output"; return 1; }
+}

tests/e2e/playwright.config.ts

@@ -29,8 +29,14 @@ export default defineConfig({
       // green suite with a test that does not exist. 17- joined on 2026-09-16
       // (the hermes vhost, whose upstream is a host address rather than a
       // container name, so it needs its own pass condition). 18- joined on
-      // 2026-09-24 (OpenWebUI SSO through authentik).
+      // 2026-09-24 (OpenWebUI SSO through authentik). 20- joined on 2026-10-21
-      testMatch: /(10|12|13|16|17|18)-.*\.spec\.ts/,
+      // (Grafana's OAuth2 relying-party login, which the edge-only 13- never
+      // follows). 21- joined 2026-10-21 (the open-notebook vhost, deployed
+      // 2026-10-19 - the newest guarded app, in neither GUARDED list nor any
+      // spec until now: the 17-hermes "uncovered vhost broke silently" class).
+      // 22- joins Prometheus's app-is-really-there proof (its own API), closing
+      // the "not a login wall but maybe a 404" hole 20- closed for Grafana.
+      testMatch: /(10|12|13|16|17|18|20|21|22)-.*\.spec\.ts/,
       use: { storageState: ".auth/qa-e2e.json" },
       dependencies: ["setup"],
     },

tests/e2e/specs/14-guard.spec.ts

@@ -15,7 +15,7 @@ const BASE = "https://www.franksdank.cloud";
 // Every vhost that Caddy routes through anubis:8923 (see dankclaw/Caddyfile).
 // Keep this list in lockstep with the Caddyfile: a new guarded vhost must be
 // added here or the guard-chain audit is incomplete.
-const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki"];
+const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki", "notebook"];
 
 // Markers that prove app content leaked past the guard to an anonymous user.
 const LEAK_MARKERS: Record<string, string[]> = {
@@ -25,6 +25,7 @@ const LEAK_MARKERS: Record<string, string[]> = {
   "seerr.franksdank.cloud": ["Request Movie"],
   "grafana.franksdank.cloud": ["Dashboards"],
   "prometheus.franksdank.cloud": ["Expression Browser", "Graph"],
+  "notebook.franksdank.cloud": ["Open Notebook"],
 };
 
 test("homepage root (/) is guarded for unauthenticated users", async ({ page }) => {

tests/e2e/specs/20-grafana-sso.spec.ts

@@ -0,0 +1,56 @@
+import { expect, test } from "@playwright/test";
+
+// Grafana (grafana.franksdank.cloud) is an OAuth2 RELYING PARTY, not an
+// outpost-fronted proxy: the guard chain (13-sso) only proves an authenticated
+// session reaches Grafana's login page. The login itself happens through
+// Grafana's own "Sign in with Authentik" button, which the edge test never
+// follows — that is exactly how the RP login could be dead (500 on authorize,
+// or an email-claim gap bouncing back to /login) while 13- still passed green.
+// This spec drives the real RP round-trip and asserts a live Grafana session.
+
+const HOST = "grafana.franksdank.cloud";
+const CHALLENGE = "Making sure you're not a bot";
+
+test("SSO: Grafana relying-party login lands a real session", async ({ page, request }) => {
+  test.setTimeout(180_000);
+
+  // Warm the guard: the vhost runs its own Anubis proof-of-work on first visit.
+  await page.goto(`https://${HOST}/`);
+  await expect
+    .poll(
+      async () => !(await page.locator("body").innerText()).includes(CHALLENGE),
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+
+  // The "Sign in with Authentik" button targets this RP start URL exactly.
+  // Going straight there is the same code path without depending on the SPA's
+  // button markup.
+  await page.goto(`https://${HOST}/login/generic_oauth`);
+
+  // The chain: grafana -> auth (implicit consent, the session is already
+  // there) -> back to grafana with the code -> grafana provisions the user and
+  // redirects to "/". A dead provider/claims config bounces back to /login.
+  await expect
+    .poll(
+      async () => {
+        const u = new URL(page.url());
+        return u.hostname === HOST && u.pathname !== "/login";
+      },
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+
+  const body = await page.locator("body").innerText();
+  expect(body).not.toContain("Login failed");
+  expect(body.toLowerCase()).not.toContain("powered by authentik"); // IdP flow shell
+  expect(body).not.toContain(CHALLENGE);
+
+  // A genuine logged-in session exists: /api/user answers 200 with an email,
+  // not 401. This is the assertion the edge-only spec could never make.
+  const resp = await page.request.get(`https://${HOST}/api/user`);
+  expect(resp.status()).toBe(200);
+  const user = await resp.json();
+  expect(typeof user.email).toBe("string");
+  expect(user.email.length).toBeGreaterThan(0);
+});

tests/e2e/specs/21-notebook.spec.ts

@@ -0,0 +1,52 @@
+import { expect, test } from "@playwright/test";
+
+// open-notebook (notebook.franksdank.cloud), deployed 2026-10-19, is a guarded
+// vhost whose outpost (notebook-proxy) fronts an app with its OWN password
+// login - the same shape as hermes (17-) and OpenWebUI (18-): the guard chain
+// (Anubis PoW -> authentik outpost) must hand off to the app, and the app's own
+// login page is the expected landing. Until now this host appeared in neither
+// the 13-/14- GUARDED lists nor any dedicated spec, so a broken upstream would
+// have stayed invisible - the exact 17-hermes "uncovered vhost" failure class.
+// 14- now covers the unauthenticated gate; this spec owns authenticated
+// reachability with a POSITIVE app marker, not just "not a login wall".
+
+const VHOST = "https://notebook.franksdank.cloud/";
+const CHALLENGE = "Making sure you're not a bot";
+
+test("notebook vhost reaches the open-notebook app through the guard chain", async ({ page }) => {
+  test.setTimeout(180_000);
+  await page.goto(VHOST);
+
+  // The Anubis proof-of-work clears in a real browser and the outpost hands off
+  // to the app; the user must never be parked on an authentik flow.
+  await expect
+    .poll(async () => !(await page.locator("body").innerText()).includes(CHALLENGE), { timeout: 90_000 })
+    .toBe(true);
+  await expect
+    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+    .toBe("notebook.franksdank.cloud");
+  expect(
+    new URL(page.url()).pathname.startsWith("/if/"),
+    "must not be parked on an authentik flow",
+  ).toBe(false);
+  expect(
+    await page.locator('input[name="uidField"]').count(),
+    "must not show the authentik login form",
+  ).toBe(0);
+
+  // Positive identity: the page really is open-notebook's own login, not a
+  // Caddy error or the IdP shell. The app answering its own login is the pass.
+  await expect(page, "destination title must be Open Notebook").toHaveTitle(/open notebook/i, {
+    timeout: 30_000,
+  });
+  await expect(page.locator("body"), "open-notebook login marker").toContainText(
+    /enter your password/i,
+    { timeout: 30_000 },
+  );
+  const body = await page.locator("body").innerText();
+  expect(body, "must not leak the IdP flow shell").not.toContain("Powered by authentik");
+  expect(body, "must not be a Caddy upstream error").not.toContain("Bad Gateway");
+
+  const resp = await page.request.get(VHOST);
+  expect(resp.status(), "open-notebook must answer, not error").toBeLessThan(500);
+});

tests/e2e/specs/22-prometheus-sso.spec.ts

@@ -0,0 +1,33 @@
+import { expect, test } from "@playwright/test";
+
+// Prometheus (prometheus.franksdank.cloud) sits behind the same embedded outpost
+// as Grafana. 13-sso and 11-admin only ever assert it is "not a login wall and
+// under 500" - which a Prometheus serving a 404 or a blank page would ALSO pass.
+// This is the app-is-really-there proof 20- provides for Grafana: with the
+// standard session the outpost hands off silently to the Prometheus UI, and the
+// app's own API answers success. A guard-route or upstream regression that lets
+// any non-Prometheus answer through fails here.
+
+test("SSO: Prometheus app answers its own API through authentik", async ({ page, request }) => {
+  test.setTimeout(180_000);
+  await page.goto("https://prometheus.franksdank.cloud/");
+
+  // Challenge clears, outpost hands off silently to the app (no credential prompt).
+  await expect
+    .poll(
+      async () => !(await page.locator("body").innerText()).includes("Making sure you're not a bot"),
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+  await expect
+    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+    .toBe("prometheus.franksdank.cloud");
+  expect(await page.locator('input[name="uidField"]').count(), "no authentik form").toBe(0);
+  await expect(page, "must land on the Prometheus UI").toHaveTitle(/prometheus/i, { timeout: 30_000 });
+
+  // Load-bearing assertion: the Prometheus API itself answers, authenticated,
+  // through the outpost. An IdP bounce, a 404, or a raw upstream error fails here.
+  const api = await request.get("https://prometheus.franksdank.cloud/api/v1/status/buildinfo");
+  expect(api.status(), "Prometheus API must answer 200 through the guard").toBe(200);
+  expect((await api.json()).status, "Prometheus API must report success").toBe("success");
+});

tests/frank_archon_unit.bats

@@ -145,7 +145,7 @@ nosudo_bin() {
     # (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
     # seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
     # tied inside the +-3% noise floor — slots buy concurrency, not speed.
-    grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_MAX_SEQS=4$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
@@ -158,7 +158,7 @@ nosudo_bin() {
     # so the pool's token count is a boot-ledger line — verified live by the
     # capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
     grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
-    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=49152$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
     # Prefix cache: pinned host RAM only (the disk tier was removed), and the
@@ -177,8 +177,11 @@ nosudo_bin() {
     [ "$ttmt" -gt "$ttmm" ]
     pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
     ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_CACHE_MIB=/{print $2}' "$ENV_FILE")
-    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
+    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host + MNBT + the auto
-    ledger=$(( 2 * pool + 48824 + ph + 2048 ))
+    # checkpoint store (MAX_SEQS x 10 slots of 112 MiB -- the store is pinned host
+    # memory too, and it grows with the slot count).
+    seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
+    ledger=$(( 2 * pool + 48824 + ph + 2048 + seqs * 10 * 112 ))
     [ "$ttmm" -ge "$ledger" ]
     # The .rad size floor is positive (a truncated convert must fail fast).
     grep -q '^FRANK_ARCHON_RAD_MIN_MIB=' "$ENV_FILE"

tests/frank_dragoon_unit.bats

@@ -3,7 +3,7 @@
 # of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
 # Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
 # digest-pinned), driven by the root unit's serve script — 2 slots over ONE
-# 200,704-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0)
+# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
 # and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
 # shape, which itself cut over the halogen 27B-dense shape on 2026-10-06.
 #
@@ -85,7 +85,7 @@ good_env() {
         "FRANK_DRAGOON_VISION_BYTES=0"
         "FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
         "HALOGEN_MODEL_ID=frank-dragoon"
-        "HALOGEN_CTX=200704"
+        "HALOGEN_CTX=262144"
         "HALOGEN_KV_POOL_POSITIONS=401408"
         "HALOGEN_KV_SLOTS=2"
         "HALOGEN_SPEC_ADAPT=0"
@@ -159,9 +159,11 @@ good_env() {
     reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
     # The pool is ONE budget behind the slots (boot ledger form: "2 slots over
     # ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
-    # a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx
+    # a per-slot private pool. Since 2026-10-21 the ceiling is the checkpoint's
-    # (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be
+    # native 262,144 while the pool stays 401,408 (raised 2026-10-18): two
-    # resident at once — frank-zealot parity on total in-flight context. What
+    # MAXIMUM harness-length (200,000-token) conversations are resident at once;
+    # two literal ceiling-length requests would queue — widening that is a pool
+    # raise, not a window move.
     # still binds: the pool must back ONE full-length request (below that the
     # advertised per-request ceiling would be unreachable, hence pool >= ctx),
     # it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
@@ -287,7 +289,7 @@ $(cat "$TEST_TMP/serve.log")"
     [[ "$output" == *"--name frank-dragoon-halogen"* ]]
     [[ "$output" == *"--publish 7770:8731"* ]]
     [[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
-    [[ "$output" == *"-e HALOGEN_CTX=200704"* ]]
+    [[ "$output" == *"-e HALOGEN_CTX=262144"* ]]
     [[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
     [[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
     [[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
@@ -306,27 +308,32 @@ $(cat "$TEST_TMP/serve.log")"
 
 @test "the shipped env keeps the harness fit law inside the ceiling" {
     # HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400
-    # naming the limit, never a silent truncation). The law is INVARIANT to
+    # naming the limit, never a silent truncation). The law: the harness
-    # the value: the harness window must sit ~45k UNDER it, because the
+    # window must sit ~45k UNDER the ceiling, because the harness clamps
-    # harness clamps max_tokens = min(45056, window − est − 4096), compacts
+    # max_tokens = min(45056, window − est − 4096), compacts at the trigger
-    # at 85 % of its declared window, and its chars/4 est UNDER-counts the
+    # percent of its declared window, and its chars/4 est UNDER-counts the
-    # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens at
+    # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens
-    # est 152,938 on 2026-10-07 — with window == ceiling, every request over
+    # at est 152,938 on 2026-10-07 — with window ≈ ceiling, every long
-    # ~65 % of the window was a guaranteed 400 before compaction could fire).
+    # request is a guaranteed 400 before compaction can fire). The 2026-10-21
-    # The shipped pair is ctx 200704 / window 155648 (lowered from
+    # pair: ceiling 262144 (native) under a harness window of 200,000.
-    # 262144/217088 on 2026-10-17); the gap must absorb the error at BOTH
+    # The declared window is read from the endpoint page — the one owner —
-    # monotone maxima of est + budget (est ≤ window − 4096):
+    # so env, models.yml and page that disagree FAIL HERE, not as a 400
-    #   full budget at the compaction threshold: 1.03*132300 + 45056
+    # mid-session (the recurring failure this seam exists for).
-    #   clamped budget at the clamp's zero point: 1.03*151552 + 0
+    grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE"
-    # and the compaction request that rides on frank-zealot (its
+    ctx="$(sed -nE 's/^HALOGEN_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
-    # 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k).
+    page="$ROOT/serverAppConfigs-docs/LLM/frank-dragoon.md"
-    grep -qE '^HALOGEN_CTX=200704$' "$ENV_FILE"
+    grep -qE 'contextWindow: 200000' "$page"
-    ctx=200704
+    hw="$(grep -oE 'contextWindow: [0-9]+' "$page" | head -1 | grep -oE '[0-9]+')"
-    hw=155648; mt=45056
+    mt=45056
-    thr=$(( hw * 85 / 100 ))
+    # Invariant checked at the live 95 % trigger (config.yml
+    # thresholdPercent) — strictly stronger than the 85 % design point.
+    # Both monotone maxima of est + budget (est ≤ window − 4096):
+    thr=$(( hw * 95 / 100 ))
     [ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
     zero=$(( hw - 4096 ))
     [ $(( zero * 103 / 100 )) -le "$ctx" ]
+    # The compaction request rides on frank-zealot's 200,704-token slot:
+    # input ≈ threshold − keepRecent(20k).
     [ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
 }
 
@@ -423,9 +430,12 @@ npu_fixture() {
     # the stock root:render 0660 mode, and the serve gate checks exactly that.
     touch "$TEST_TMP/accel0"
     chmod 666 "$TEST_TMP/accel0"
-    mkdir -p "$TEST_TMP/xrt"
+    # basename carries a multiarch shape: the third mount prefix is derived
+    # from the XRT dir's basename (the NPU child searches
+    # /opt/xilinx/lib/<multiarch>/, measured 2026-10-21).
+    mkdir -p "$TEST_TMP/x86_64-linux-gnu"
     for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
-        touch "$TEST_TMP/xrt/$l"
+        touch "$TEST_TMP/x86_64-linux-gnu/$l"
     done
     mkdir -p "$TEST_TMP/npu/qwen3-embedding-0.6b/devices"
     mkdir -p "$TEST_TMP/sys/class/drm/card1/device"
@@ -435,7 +445,7 @@ npu_fixture() {
         "HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"
         "FRANK_DRAGOON_NPU_DIR=$TEST_TMP/npu"
         "FRANK_DRAGOON_DEV_ACCEL=$TEST_TMP/accel0"
-        "FRANK_DRAGOON_XRT_DIR=$TEST_TMP/xrt"
+        "FRANK_DRAGOON_XRT_DIR=$TEST_TMP/x86_64-linux-gnu"
         "FRANK_DRAGOON_SYSFS=$TEST_TMP/sys"
     )
 }
@@ -465,11 +475,15 @@ $(cat "$TEST_TMP/serve.log")"
     ! grep -q 'accel0' "$TEST_TMP/serve.log"
 }
 
-@test "the lane arms the accel node, the staged model dir and each XRT library twice" {
+@test "the lane arms the accel node, the staged model dir and each XRT library three times" {
-    # Each library mounts at the image's XRT path AND at its own path: the
+    # Each library mounts at the image's XRT path, at its own path (the distro
-    # distro libraries find each other where they were installed, and an
+    # libraries find each other where they were installed), and at
-    # /opt/xilinx/xrt of symlinks into the system dir dangles inside the
+    # /opt/xilinx/lib/<multiarch>/ — the prefix the NPU child searches and
-    # container (the vendor calls that out by name).
+    # names when it is missing (`No such library
+    # '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'`, measured
+    # 2026-10-21: the lane died and the supervisor took the whole container
+    # with it, 88-restart loop). An /opt/xilinx/xrt of symlinks into the
+    # system dir dangles inside the container (the vendor calls that out).
     podman_stub
     good_env
     npu_fixture
@@ -484,8 +498,9 @@ $(cat "$TEST_TMP/serve.log")"
     [[ "$output" != *":/models/npu"* ]]
     [[ "$output" == *"-e HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"* ]]
     for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
-        [[ "$output" == *"-v $TEST_TMP/xrt/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
-        [[ "$output" == *"-v $TEST_TMP/xrt/$l:$TEST_TMP/xrt/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:$TEST_TMP/x86_64-linux-gnu/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/lib/x86_64-linux-gnu/$l:ro"* ]]
     done
     # The Flash payload still rides the same container: the lane is additive.
     [[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
@@ -560,7 +575,7 @@ $(cat "$TEST_TMP/serve.log")"
     podman_stub
     good_env
     npu_fixture
-    rm "$TEST_TMP/xrt/libxrt_driver_xdna.so.2"
+    rm "$TEST_TMP/x86_64-linux-gnu/libxrt_driver_xdna.so.2"
     run_with_log
     [ "$status" -ne 0 ]
     [[ "$output" == *"libxrt_driver_xdna.so.2 missing"* ]]

tests/frankbook_quadlet.bats

@@ -77,6 +77,7 @@ FRANKBOOK_UNITS=(
 }
 
 @test "readarr keeps its pinned image" {
-    grep -q '^Image=lscr.io/linuxserver/readarr:amd64-1.0.980$' \
+    # 2026-10-21 fleet law: version tag AND digest (tests/quadlet_image_pins.bats)
+    grep -qE '^Image=lscr\.io/linuxserver/readarr:amd64-1\.0\.980@sha256:[0-9a-f]{64}$' \
         "$REPO_ROOT/frankbook-pro/readarr/quadlet/readarr.container" || return 1
 }