docs(comfyui): the GTT aperture is not free RAM beside frank-dragoon

Measured 2026-10-21 with frank-dragoon live: MemAvailable 15.3 GiB,
iGPU busy 1.75 GiB. The Qwen-Image weight sets (13.3-14.2 GB) do not
fit that budget, and the page's 122 GB /system_stats figure is the
aperture, not headroom. Adds the fit table and the
stop-mask-render-verify procedure (the 3090 bench law, applied to the
APU); ComfyUI.md gotcha links it.

AGENTS.md

@@ -176,11 +176,12 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
 (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
 IP, the one bind a container can reach as `host.containers.internal`;
 iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
-Prometheus targets read down for structural reasons (HTML-scraped "probes",
+The former structural target downs (HTML-scraped "probes", Gitea metrics off)
-Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
+are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
-a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
+(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
-without `--metrics`. See
+every Prometheus job reports up. `frank-zealot` was down for a different reason
-`serverAppConfigs-docs/Infrastructure/Observability.md`.
+until 2026-10-18: llama.cpp will not serve `/metrics` at all without
+`--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.
 
 ## Testing
 
@@ -419,16 +420,17 @@ disagrees with it is the bug.
   digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
   Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
   `/home/dank/logs/frank_dragoon.log`, NOT the journal.
-- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
+- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
-  is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
+  **per-request ceiling** (a longer prompt is a 400, never a truncation);
-  `HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
+  `HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
-  the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
+  pool positions buy concurrency, the per-request window never moves;
-  drafter always armed (the adaptive default disarms it and halves decode);
+  `HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
-  server sampler 1.0/0.95/20.
+  default disarms it and halves decode); server sampler 1.0/0.95/20.
-- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
+- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
-  The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
+  ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
-  under-counts, so a harness window equal to the ceiling makes every long
+  estimate under-counts (trigger is 95 %), so a harness window at the ceiling
-  request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
+  makes every long request a guaranteed 400.
+  Reasoning arrives in `reasoning_content`; the
   template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
   OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
   `/completion`, `/ping`, `/props`, `/tokenize` are 404.

dankclaw/gitea/quadlet/gitea-server.container

@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
 Environment=DATABASE_PORT=5432
 Environment=USER_UID=1000
 Environment=USER_GID=1000
+# Prometheus scrapes /metrics over the published LAN port (job "gitea" in
+# shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
+# permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
+# the container; the only route in is this host's published 3000, and the
+# exposed surface is counters only — same posture as the LLM /metrics.
+Environment=GITEA__metrics__ENABLED=true
+Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
 PublishPort=0.0.0.0:3000:3000
 PublishPort=2222:22
 Volume=/home/dank/Containers/gitea/data:/data

dankclaw/observability/quadlet/blackbox-exporter.container

@@ -0,0 +1,35 @@
+# Quadlet unit: blackbox exporter (rootless, boot-persistent).
+#
+# Install path on dankclaw:
+#   ~/.config/containers/systemd/blackbox-exporter.container
+#
+# WHY Network=host. The probes it runs are (a) the host's own LAN IP
+# (192.168.0.143:<port>) and (b) container names on dankclaw_default (comfyui).
+# A bridge container cannot reach the host's LAN IP at an unpublished port
+# (netavark DNATs published ports only — see Container Host Reachability in the
+# vault); a host-netns process reaches both. Prometheus (bridge) reaches THIS
+# container the same way it reaches the signal relay: the quadlet-published
+# host.containers.internal address (169.254.1.2), mapped via HostAddr= on the
+# prometheus unit.
+#
+# The 9115 metrics port publishes on loopback only — it is scraped by the
+# prometheus container through the gateway, and never opened on the LAN.
+
+[Unit]
+Description=Blackbox exporter (HTTP probe prober)
+After=network-online.target
+
+[Container]
+ContainerName=blackbox-exporter
+Exec=/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/config.yml
+Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+HealthCmd=/bin/sh -c "wget -q -O /dev/null http://127.0.0.1:9115/-/healthy || exit 1"
+HealthInterval=30s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/cadvisor.container

@@ -0,0 +1,56 @@
+# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKCLAW.
+#
+# Install path on dankclaw: ~/.config/containers/systemd/cadvisor.container
+# Unit name: cadvisor.service (container name: cadvisor)
+#
+# WHY THIS EXISTS. The app stack is ~34 rootless Podman containers and until
+# now nothing reported per-container CPU/memory — Grafana could only see the
+# host (node_exporter) and the LLM engines. cadvisor reads the host cgroup
+# tree and exports container_cpu_* / container_memory_* per container.
+#
+# THE LOAD-BEARING FLAG IS --cgroupns=host. Podman starts every container in a
+# PRIVATE cgroup namespace, so an exporter container sees only its OWN cgroup
+# subtree: cadvisor exported a single root series until this flag was added,
+# then it walked the whole tree (996 per-container series measured 2026-10-20).
+# The quadlet unit name is carried in the cgroup path — app.slice/<name>.service/
+# libpod-payload-<hex> — so Prometheus relabels that path into a `container`
+# label (shared/prometheus/prometheus.yml, cadvisor-dankclaw job) and no
+# libpod/podman socket is needed to name containers.
+#
+# BINDING. cadvisor has no per-interface listen flag (unlike node_exporter's
+# --web.listen-address), so it binds 0.0.0.0 INSIDE its own network namespace.
+# The bridge PublishPort is what keeps the unauthenticated endpoint off the
+# host's other interfaces: it publishes exactly loopback (host-local probes and
+# the deployer's health gate) and this host's LAN address (Prometheus, a bridge
+# container, reaches a host-published port at its LAN address — the same netavark
+# rule node-exporter relies on, only here the port really is published).
+#
+# It is rootless on purpose: dank owns the app containers' cgroups, so a
+# rootless cadvisor reads every one of them; root would buy nothing here.
+
+[Unit]
+Description=cAdvisor container metrics (Dankclaw)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=cadvisor
+Image=gcr.io/cadvisor/cadvisor:v0.49.1
+PodmanArgs=--pid=host --cgroupns=host
+PublishPort=127.0.0.1:8080:8080
+PublishPort=192.168.0.143:8080:8080
+Volume=/sys:/sys:ro
+Volume=/:/rootfs:ro
+Exec=--housekeeping_interval=30s
+HealthCmd=wget --spider -q http://localhost:8080/healthz
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=40s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/grafana.container

@@ -47,9 +47,24 @@ Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
 # client_id query parameter — /application/o/<slug>/authorize/ does not exist.
 Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/
 Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/
-# ?api=1 = introspection-style userinfo: returns custom claims (groups) too.
+# Standard OIDC userinfo. Grafana v13 fetches <api_url>/emails for the address
-Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/?api=1
+# ONLY when the token carries no `email` claim (authentik has no /emails path and
-Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=false
+# 404s it, failing the login) — so the claim must actually be there: the fix is
+# the grafana-email-scope mapping (see scripts/authentik-grafana-sso-fix.sh),
+# not a Grafana flag. Do NOT append ?api=1 to this URL: Grafana then GETs
+# <...>/userinfo/?api=1/emails, a 403.
+Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/
+# authentik's scopes are openid/email/profile. `user:email` (a GitLab-ism) is
+# not a known scope here; requesting it makes the authorize view crash.
+# Quoted: quadlet word-splits Environment=, so an unquoted spaced list
+# truncates SCOPES to its first token.
+Environment="GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile"
+Environment=GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
+Environment=GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=sub
+Environment=GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=preferred_username
+# First SSO login auto-provisions the user. Safe: unauthenticated traffic never
+# reaches this callback — the Anubis -> Authentik guard fronts every request.
+Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
 Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true
 Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true
 Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1

dankclaw/observability/quadlet/prometheus.container

@@ -45,6 +45,11 @@ Wants=network-online.target
 ContainerName=prometheus
 Image=docker.io/prom/prometheus:latest
 Network=dankclaw_default
+# Name the podman gateway so scrape jobs can reach host-netns processes (the
+# blackbox exporter listens on the host's loopback; a bridge container reaches
+# it at this address, never at 127.0.0.1). Same mechanism the hermes Caddy
+# upstream learned the hard way (Container Host Reachability in the vault).
+HostAddr=host.containers.internal=169.254.1.2
 User=0:0
 Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
 PublishPort=127.0.0.1:9090:9090

dankserver/observability/quadlet/cadvisor.container

@@ -0,0 +1,39 @@
+# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKSERVER.
+#
+# Install path on dankserver: ~/.config/containers/systemd/cadvisor.container
+# Unit name: cadvisor.service (container name: cadvisor)
+#
+# The Dankserver twin of dankclaw/observability/quadlet/cadvisor.container.
+# Same rootless shape, same --cgroupns=host requirement (see that file for the
+# full story), only the LAN address differs: this host publishes on
+# 192.168.0.235:8080 so Prometheus on Dankclaw can scrape it over the real
+# network (cross-host, so the host.containers.internal trick does not apply).
+# It reports the media/library stack (romm, plex, jellyfin, copyparty, calibre,
+# tautulli) by their quadlet unit names via the cgroup path.
+
+[Unit]
+Description=cAdvisor container metrics (Dankserver)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=cadvisor
+Image=gcr.io/cadvisor/cadvisor:v0.49.1
+PodmanArgs=--pid=host --cgroupns=host
+PublishPort=127.0.0.1:8080:8080
+PublishPort=192.168.0.235:8080:8080
+Volume=/sys:/sys:ro
+Volume=/:/rootfs:ro
+Exec=--housekeeping_interval=30s
+HealthCmd=wget --spider -q http://localhost:8080/healthz
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=40s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankserver/plex/quadlet/plex.container

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Container]
-Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
+Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329
 ContainerName=plex
 Network=dankserver_appnet:alias=plex
 Environment=PUID=0

scripts/authentik-grafana-sso-fix.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Repair the Grafana OAuth2 relying-party SSO on Dankclaw's authentik.
+#
+# grafana.franksdank.cloud logs in through authentik as an OAuth2 relying party.
+# Two DB-side facts (the provider was created by hand via `ak shell`, never
+# repo-managed) silently broke that login, and BOTH had previously been
+# mis-recorded here as a single "the property-mapping evaluator is broken, so
+# userinfo returns only `sub`" myth. They are neither, and the evaluator is fine:
+#
+#   1. provider.authorization_flow was NULL. The authorize view builds a flow
+#      planner from it and crashes with
+#      `AttributeError: 'NoneType' object has no attribute 'slug'` — an HTTP 500
+#      "Server Error — Powered by authentik", before any claim is touched.
+#      Fix: point it at default-provider-authorization-implicit-consent, the
+#      flow every other provider in this instance already uses.
+#   2. The `grafana-email-scope` ScopeMapping's expression was the bare string
+#      `user.email`. authentik merges a scope mapping's result into the token as
+#      a MAPPING: a returned dict adds its keys as claims; a returned string is
+#      keyed under the mapping's NAME ("grafana-email-scope"), so no `email`
+#      claim is ever emitted. Grafana, finding no email, falls back to
+#      GET <api_url>/emails, authentik 404s that path, and the login dies with
+#      "Login failed: InternalError". Fix: return
+#      {"email": user.email, "email_verified": True} — the shape the stock
+#      OpenID 'email' mapping uses. The dict-valued `grafana-profile-scope`
+#      mapping has always worked, which is the proof the evaluator is fine.
+#
+# This is authentik DB state. It is NOT in any deployable file: a blueprint
+# re-import, or an ak-shell re-creation of the grafana provider, reverts BOTH
+# facts and the symptom returns verbatim. Re-run this after any such rebuild.
+# Every step is idempotent.
+#
+# Grafana's own half — SCOPES="openid email profile", a standard userinfo API_URL
+# with no `?api=1`, and ALLOW_SIGN_UP=true — lives in
+# dankclaw/observability/quadlet/grafana.container and deploys normally.
+#
+# Usage: authentik-grafana-sso-fix.sh          (override host: AUTHENTIK_HOST=…)
+set -euo pipefail
+HOST="${AUTHENTIK_HOST:-dankclaw}"
+
+ssh "$HOST" 'podman exec -i authentik-server ak shell' <<'PYEOF'
+from authentik.flows.models import Flow
+from authentik.providers.oauth2.models import OAuth2Provider
+
+p = OAuth2Provider.objects.filter(name="grafana").first()
+if not p:
+    raise SystemExit("no OAuth2Provider named 'grafana'")
+
+# 1. The authorization flow every other provider uses.
+want = Flow.objects.filter(slug="default-provider-authorization-implicit-consent").first()
+if not want:
+    raise SystemExit("default-provider-authorization-implicit-consent flow is missing")
+if p.authorization_flow_id != want.pk:
+    p.authorization_flow = want
+    p.save()
+    print(f"set grafana.authorization_flow -> {want.slug}")
+else:
+    print(f"grafana.authorization_flow already {want.slug}")
+
+# 2. The email scope mapping must RETURN A DICT, not a bare string.
+m = p.property_mappings.filter(name="grafana-email-scope").first()
+if not m:
+    raise SystemExit("grafana-email-scope mapping is missing")
+fixed = 'return {"email": user.email, "email_verified": True}'
+if m.expression.strip() != fixed:
+    m.expression = fixed
+    m.save()
+    print("repaired grafana-email-scope -> returns {email, email_verified}")
+else:
+    print("grafana-email-scope expression already correct")
+
+# 3. The id_token must carry the scope claims; Grafana reads identity from it.
+if not p.include_claims_in_id_token:
+    p.include_claims_in_id_token = True
+    p.save()
+    print("enabled include_claims_in_id_token")
+else:
+    print("include_claims_in_id_token already on")
+PYEOF
+
+echo "Done. Verify: sign in at https://grafana.franksdank.cloud via Authentik;"
+echo "the round trip must land on / and /api/user must answer 200 with an email."

scripts/deploy-plan.py

@@ -34,12 +34,16 @@ SERVICE_ORDER = (
     "hermes",
     "hermes-dashboard",
     "hermes-webui",
-    "prometheus",
     "grafana",
     "loki",
     "alloy",
     "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
     "alertmanager",
+    # The prober must answer before prometheus re-runs the services job.
+    "blackbox-exporter",
+    "prometheus",
     "authentik-postgres",
     "authentik-server",
     "authentik-worker",
@@ -92,7 +96,10 @@ DANKCLAW_SERVICES = {
     "loki",
     "alloy",
     "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
     "alertmanager",
+    "blackbox-exporter",
     "authentik-postgres",
     "authentik-server",
     "authentik-worker",
@@ -124,6 +131,8 @@ DANKSERVER_SERVICES = {
     "readarr",
     "frank-archon",
     "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
 }
 
 # FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
@@ -271,6 +280,8 @@ def services_for_path(path: str, target: str) -> set[str]:
             # Provisioned datasources and dashboards: git is the source of
             # truth, so a change here must restart the service that reads them.
             _add(services, "grafana")
+        elif group == "blackbox":
+            _add(services, "blackbox-exporter")
         elif group == "homepage":
             # Homepage is a Dankclaw-owned deployment. Shared homepage files
             # are intentionally not copied during a Dankserver-only run.
@@ -374,7 +385,7 @@ def services_for_path(path: str, target: str) -> set[str]:
             # consumer (prometheus before grafana, loki before alloy).
             if len(parts) >= 4 and parts[2] == "quadlet":
                 name = parts[3].removesuffix(".container")
-                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
+                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
                     _add(services, name)
             return services
         if len(parts) >= 2 and parts[1] == "gitea":
@@ -417,8 +428,8 @@ def services_for_path(path: str, target: str) -> set[str]:
             # node-dankserver job had nothing to scrape.
             if len(parts) >= 4 and parts[2] == "quadlet":
                 name = parts[3].removesuffix(".container")
-                if name == "node-exporter":
+                if name in {"node-exporter", "cadvisor"}:
-                    _add(services, "node-exporter")
+                    _add(services, name)
             return services
         # frank-archon's quadlet-era directory left the tree on 2026-09-26
         # (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
@@ -449,6 +460,12 @@ def services_for_path(path: str, target: str) -> set[str]:
             # change to it redeploying alertmanager is the only path that
             # reaches it on the host.
             _add(services, "alertmanager")
+        if name in {"gpu-exporter.py", "gpu-exporter.service"}:
+            # The host-process GPU exporter is installed directly by the
+            # gpu-exporter deploy on BOTH hosts, so its script and unit select
+            # that service (target_services then keeps it only on the host being
+            # planned). Editing the collector restarts the running exporter.
+            _add(services, "gpu-exporter")
         if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
             # llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
             # files here; each one selects frank-zealot only. The ik_llama.cpp

scripts/deploy-remote.sh

@@ -86,7 +86,7 @@ service_is_allowed() {
         dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
         dankclaw:comfyui|dankclaw:hermes|\
         dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:alertmanager|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|\
         dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
         dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -97,7 +97,7 @@ service_is_allowed() {
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
         dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
-        dankserver:frank-archon|dankserver:node-exporter|\
+        dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
         frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
         frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
         frankbook-pro:readarr)
@@ -275,14 +275,81 @@ install_signal_relay() {
     return 1
 }
 
+# Host-process GPU metrics exporter (amdgpu sysfs + nvidia-smi). Not a quadlet:
+# it runs python3 from the checkout on the host so it can read the DRM/cgroup
+# sysfs and shell out to nvidia-smi. Same two-address bind posture as
+# node-exporter/cadvisor (loopback + this host's LAN), so it is never a wildcard.
+deploy_gpu_exporter() {
+    local unit_dir="$HOME/.config/systemd/user" lan dropin i
+    case "$TARGET" in
+        dankclaw) lan=192.168.0.143 ;;
+        dankserver) lan=192.168.0.235 ;;
+        *) echo "gpu-exporter: unknown target '$TARGET'" >&2; return 2 ;;
+    esac
+    mkdir -p "$unit_dir"
+    cp scripts/gpu-exporter.service "$unit_dir/gpu-exporter.service"
+    dropin="$unit_dir/gpu-exporter.service.d"
+    mkdir -p "$dropin"
+    printf '[Service]\nEnvironment=GPU_EXPORTER_BINDS=127.0.0.1,%s\n' "$lan" \
+        > "$dropin/bind.conf"
+    systemctl --user daemon-reload
+    systemctl --user enable gpu-exporter.service
+    systemctl --user restart gpu-exporter.service
+    # The interpreter needs a moment to bind after restart (the same race that
+    # once failed a healthy signal-relay deploy).
+    for i in $(seq 1 15); do
+        curl -fsS -m 5 http://127.0.0.1:9401/metrics >/dev/null && return 0
+        sleep 1
+    done
+    echo "gpu-exporter is not answering on 127.0.0.1:9401/metrics" >&2
+    return 1
+}
+
 copy_loki_config() {
     run_privileged mkdir -p /opt/loki/config
     run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
 }
 
 copy_alloy_config() {
-    run_privileged mkdir -p /opt/alloy/config
+    # The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
-    run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
+    # carries the host's name, so Loki streams say WHICH machine's journal a
+    # line came from without a second config file per host.
+    # *_CONF_ROOT are test seams: the bats cases execute these copiers for
+    # real against a sandbox with a stubbed run_privileged.
+    local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
+    tmp="$(mktemp)"
+    sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
+    if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
+        echo "alloy config: host-label substitution failed" >&2
+        rm -f "$tmp"
+        return 1
+    fi
+    # Syntax gate: alloy's config language has no schema — a misplaced
+    # attribute (e.g. `on_error` outside the json stage) parses only at RUN
+    # time, and the unit then crash-loops with a healthy-looking deploy.
+    # Validate against the very image that will run it, before installing.
+    # Skipped only in the bats sandbox (no image there).
+    if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
+        if ! podman run --rm -v "$tmp:/c.alloy:ro" \
+                --entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
+                validate /c.alloy; then
+            echo "alloy config: validate FAILED against the pinned image" >&2
+            rm -f "$tmp"
+            return 1
+        fi
+    fi
+    run_privileged mkdir -p "$root/config"
+    run_privileged cp "$tmp" "$root/config/config.alloy"
+    rm -f "$tmp"
+}
+
+copy_blackbox_config() {
+    # Single-file mount target: the unit binds this exact path into
+    # /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
+    # BLACKBOX_CONF_ROOT is the same test seam.
+    local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
+    run_privileged mkdir -p "$root"
+    run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
 }
 
 copy_grafana_provisioning() {
@@ -291,14 +358,25 @@ copy_grafana_provisioning() {
     # datasource would be invisible to review, lost with the data volume, and
     # unknown to anyone reading the repository. The unit already mounts
     # /opt/grafana/provisioning read-only, so this only has to populate it.
-    run_privileged mkdir -p /opt/grafana/provisioning/datasources \
+    # GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
-        /opt/grafana/provisioning/dashboards
+    # copy runs unprivileged (the bats case executes this function for real).
+    local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
+    run_privileged mkdir -p "$root/provisioning/datasources" \
+        "$root/provisioning/dashboards"
     run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
-        /opt/grafana/provisioning/datasources/prometheus.yml
+        "$root/provisioning/datasources/prometheus.yml"
     run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
-        /opt/grafana/provisioning/dashboards/provider.yml
+        "$root/provisioning/dashboards/provider.yml"
-    run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
+    # EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
-        /opt/grafana/provisioning/dashboards/llm-fleet.json
+    # turns each file in the directory into a dashboard, so a copier that names
+    # files one by one silently drops new boards (measured 2026-10-21: the
+    # containers board existed in the repo since its commit and was NEVER
+    # provisioned — the panel-census browser check could not find it while the
+    # hand-placed older boards happened to be on disk). Same rule as the quadlet
+    # tier: the directory is the contract, the copy is generated state.
+    for dash in shared/grafana/provisioning/dashboards/*.json; do
+        run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
+    done
 }
 
 render_caddyfile() {
@@ -520,6 +598,11 @@ deploy_observability() {
             ;;
         node-exporter)
             ;;
+        cadvisor)
+            ;;
+        blackbox-exporter)
+            copy_blackbox_config
+            ;;
         *)
             echo "unknown observability service: $service" >&2
             return 2
@@ -533,6 +616,8 @@ deploy_observability() {
         grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
         loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
         node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
+        cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
+        blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
     esac
 }
 
@@ -1368,9 +1453,10 @@ deploy_anubis() {
 # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
 # qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
 # + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
-# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
+# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
-# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
+# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
-# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
+# run a 200,000 window with its ~45k fit gap; the pool still carries two
+# 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
 # ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
 # relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
 # file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
@@ -2000,7 +2086,8 @@ deploy_service() {
         plex) deploy_plex ;;
         tautulli) deploy_tautulli ;;
         alloy) deploy_alloy ;;
-        prometheus|grafana|loki|node-exporter|alertmanager) deploy_observability "$service" ;;
+        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
+        gpu-exporter) deploy_gpu_exporter ;;
         comfyui) deploy_comfyui ;;
         *) deploy_compose_service "$service" ;;
     esac

scripts/deploy.sh

@@ -133,13 +133,15 @@ fi
 valid_service() {
     local service="$1"
     case "$TARGET:$service" in
+        # dankclaw:searxng-network is client-only: a network quadlet with no
+        # remote dispatch handler, so the remote gate does not (must not) list it.
         dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
         dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
         dankclaw:gitea-server|dankclaw:homepage|\
         dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
         dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
         dankclaw:alertmanager|\
-        dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
+        dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
         dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
         dankclaw:open-notebook|\
         dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
         dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
         dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
         dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
-        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
+        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
-        dankserver:frank-archon|\
+        dankserver:prowlarr|dankserver:frank-archon|\
         frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
         frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
         frankbook-pro:readarr)

scripts/frank-archon.env

@@ -46,31 +46,35 @@ FRANK_ARCHON_TP=2
 # stays EXACT for single-sequence decode and any step of <= 25 rows — see
 # MAX_SEQS x (1 + SPEC_TOKENS) below.
 FRANK_ARCHON_TP_WIRE=wht6
-# Decode slots. A slot costs KV, never decode; the pool is sized to this
+# Decode slots. A slot costs KV and pinned RAM, never decode: the cache target is
-# contract and nothing more. At 1.8.1 the budget resolver caps the cache at
+# exactly MAX_SEQS x MAX_MODEL_LEN and the checkpoint store asks (1 + what the
-# 556,348 fp8 tokens — 2.8 sessions at 200,000 — so a THIRD full-height
+# retention policy keeps) per slot. Ordered 2026-10-20: FOUR slots over a pool of at
-# conversation answers 400 while /health and every shorter chat work fine
+# least 700,000 fp8 tokens, rounding UP if the resolver cannot land it exactly. The
-# (3 x 50k streams run at 77.6 t/s aggregate). Frank's trade: declare the two
+# target is therefore 4 x 200,000 = 800,000 tokens and HOST_POOL_MIB decides whether
-# slots the engine can hold and let the freed ~1.1 GiB go to the expert plane,
+# the cache gets it (expert_vram_floor, core/place/planner.cpp:100-104: the card must
-# instead of advertising a third slot that only exists at shorter prompts.
+# keep expert-plane-minus-host-room). The 2-slot shape was the answer to a
-FRANK_ARCHON_MAX_SEQS=2
+# 365,204-token pool that backed 1.8 full-height sessions; at this pool the third and
+# fourth conversations are backed by the cache instead of over-committing it.
+FRANK_ARCHON_MAX_SEQS=4
 # Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
 # limit, and models.yml contextWindow MUST quote this number (over-length is a
 # refusal, not a truncation). The pool is sized for seqs x maxlen, not one
 # sequence.
 FRANK_ARCHON_MAX_MODEL_LEN=200000
 FRANK_ARCHON_PLACEMENT=expert_tiered
-# Pinned host pool for expert rows that do not fit VRAM. Under the budget
+# Pinned host pool for expert rows that do not fit VRAM, PINNED ONCE PER RANK: the
-# resolver this is ALSO the lever that decides which experts stay on the card:
+# host must pin tp x this number, and the refusal that names it is
-# the card's leftover goes to the expert plane first and the KV cache takes the
+# core/engine_bringup.cpp:250-265. Under the budget resolver this is ALSO the KV
-# rest, so growing the pool moves experts off the card and shrinking it spills
+# lever -- the card keeps expert-plane-minus-this-room, so every MiB added moves one
-# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
+# MiB of experts off the card and into the cache. Derived from the boot ledger (a
-# refused a boot). The ledger's coverage line is the referee, not this file.
+# 23.73 GiB floor at 44,928 room puts plane + 2 units at 67.60 GiB a rank), 49,152
-# Measured, not guessed: 45,568 (PR #30) moved host_pool to 45.09 GiB and left
+# drops the floor to 19.6 GiB and leaves the cache ~6.9 GiB, which is enough to back
-# vram_kv byte-identical at 4.20 GiB / 556,348 tokens on both ranks — extra
+# the whole 4 x 200,000 target; the leftover returns to the experts. Cost is paid
-# host RAM above what the expert plane's footprint needs buys no cache. Back at
+# twice over: pinned RAM (tp x 48 GiB, against the TTM target below) and expert
-# 44,928, the size that boots clean.
+# residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
-FRANK_ARCHON_HOST_POOL_MIB=44928
+# x8 link when the router picks it. The ledger's coverage line is the referee, not
+# this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
+FRANK_ARCHON_HOST_POOL_MIB=49152
 # VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
 # floor is built from it.
 FRANK_ARCHON_GPU_HEADROOM_MIB=96
@@ -93,16 +97,29 @@ FRANK_ARCHON_SPEC_TOKENS=2
 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
 # Step sharing. A prefill chunk that shares a step with decoding sequences holds
 # the device for the chunk's whole time, and every decode row in that step waits
-# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
+# that long for its next token: measured at this shape, decode-only steps 15.5 ms
-# `auto` leaves each decode row behind the chunk a row plus its draft window and
+# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s
-# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
+# while another session prefills (law and census: [[frank-archon]]).
-# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
+# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the
-# chunk took the step and admission never ran. Interleave inserts N decode-only
+# draft window per decoding sequence (3 tokens) and an eighth of the budget per
-# steps after a chunk step taken beside decoders: it protects their cadence and
+# waiting request or waiting prefill, and the source states it will not halve for
-# holds a waiting request out of those steps, so it stays 0 until the post-migration
+# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk
-# decode-vs-prefill buckets say it is wanted.
+# steps. A number is the only value that applies whenever the chunk shares a step,
+# and it must not go under the prefill stager's smallest pass -- 1025 rows here,
+# named on the boot line "prefill staging: ... a step of more than 1024 tokens" --
+# because below it every routed layer reads its pooled experts synchronously, so
+# 512 is a trap and the vendor measures the whole job slower there. Interleave
+# inserts N decode-only steps after a chunk step taken beside decoders: it protects
+# their cadence and holds a waiting request out of those steps. That is the decoder
+# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
+# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
+# price falls on the prefilling prompt (its chunks lose every held step) and on any
+# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
+# ALONE, so it cannot buy this back while decoders run. Meter:
+# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
+# before N rises.
 FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
-FRANK_ARCHON_PREFILL_INTERLEAVE=0
+FRANK_ARCHON_PREFILL_INTERLEAVE=1
 # Prefix cache: pinned host memory only. The disk tier was removed outright
 # (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
 # be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
@@ -118,9 +135,10 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
 # RESETS pages_limit to half of RAM on every boot; below the plan the load dies
 # mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
 # itself before refusing by name, so recovery is a boot, not a visit.
-# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT
+# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
-# per the bats ledger test; at the 44,928 pool that computes to 148,920 MiB.
+# auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
-FRANK_ARCHON_TTM_MIN_MIB=150000
+# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
+FRANK_ARCHON_TTM_MIN_MIB=162000
 FRANK_ARCHON_TTM_TARGET_MIB=204800
 # Escape hatch for experiments. Unset -> no extra argv; the serve script expands
 # this unquoted on purpose, so keep multi-flag values as one string.

scripts/frank-dragoon.env

@@ -49,14 +49,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
 # stays at the image default, which is what was qualified) ------------------
 # Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
 HALOGEN_MODEL_ID=frank-dragoon
-# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
+# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
-# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
+# longer prompt is a 400 naming the limit, never a truncation; stretching
-# the page) — a window equal to the ceiling makes every long request a 400.
+# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
-HALOGEN_CTX=200704
+# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
-# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
+# page): with the window at 200,000 the gap is 62,144, which absorbs the
-# at once. Pool positions buy concurrency; the per-request window never moves
+# chars/4 under-count. A window at or near the ceiling makes every long
-# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
+# request a guaranteed 400.
-# RAM the pool takes is RAM the n-gram table's page cache loses.
+HALOGEN_CTX=262144
+# ONE pool behind both slots. The harness window is 200,000, so 401,408
+# still carries TWO maximum harness-length conversations at once
+# (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
+# ceiling-length requests would queue instead — buying those means raising
+# the pool, and RAM the pool takes is RAM the n-gram table's page cache
+# loses. Pool positions buy concurrency; the per-request window never moves
+# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
 HALOGEN_KV_POOL_POSITIONS=401408
 # Two concurrent conversations: the MTP drafter is not slot-locked on this
 # engine, so the second slot is capacity, not a speculation tax.
@@ -107,10 +114,13 @@ HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
 FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
 # Node paths and the XRT directory are variables so the bats seam can drive
 # these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
-# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
+# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
-# once at its own path, because those libraries find each other at their install
+# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
-# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
+# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
-# system dir, which dangle inside the container.
+# searches that third prefix; without it the lane dies naming it and the
+# supervisor takes the container down — 2026-10-21 boot loop). The basename of
+# this directory IS the multiarch name. The vendor forbids mounting an
+# /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
 FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
 FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
 # sysfs root the fabric-clock preflight reads (same knob the vendor's

scripts/frank-dragoon.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
+Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target
 

scripts/gpu-exporter.py

@@ -0,0 +1,212 @@
+#!/usr/bin/env python3
+"""gpu-exporter: unified per-GPU metrics for a mixed AMD/NVIDIA host.
+
+Emits the `gpu_*` Prometheus families the host-system dashboard graphs:
+
+    gpu_utilization_percent  gpu_memory_used_bytes  gpu_memory_total_bytes
+    gpu_temperature_celsius  gpu_power_watts
+
+each labelled {gpu, vendor, model}. The scrape job adds `host`.
+
+Two sources, both host-native — no container, no GPU passthrough:
+
+  * AMD (R9700s, the 8060S iGPU, the Raphael display GPU): the amdgpu sysfs tree.
+    Utilization comes ONLY from gpu_busy_percent — amd-smi reports gfx_busy_inst
+    as N/A on gfx1201 (verified 2026-10-20). VRAM is already bytes, temperature
+    is milli-degrees, power is micro-watts. Crucially power1_average is EMPTY on
+    the R9700s, so those cards legitimately have no gpu_power_watts line; a
+    missing file becomes an ABSENT metric, never a fabricated 0 that a dashboard
+    would plot as a real reading.
+
+  * NVIDIA (the RTX 3090 on Dankclaw): one `nvidia-smi --query-gpu` CSV. On
+    Dankserver there is an nvidia-smi binary but no NVIDIA GPU, so the command
+    fails; that is tolerated and simply yields no NVIDIA series.
+
+Default mode prints one scrape to stdout and exits (the test harness, and any
+`service ... --oneshot` one-off). `--serve` binds the HTTP endpoint for
+Prometheus; it binds EXPLICITLY (loopback + the host LAN address), never a
+wildcard, matching node-exporter's posture on these boxes.
+"""
+import argparse
+import glob
+import os
+import subprocess
+import sys
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+FAMILIES = [
+    ("gpu_utilization_percent", "gauge", "GPU busy time as a percentage of wall time."),
+    ("gpu_memory_used_bytes", "gauge", "Device memory in use, in bytes."),
+    ("gpu_memory_total_bytes", "gauge", "Device memory capacity, in bytes."),
+    ("gpu_temperature_celsius", "gauge", "Package temperature in degrees Celsius."),
+    ("gpu_power_watts", "gauge", "Board power draw in watts."),
+]
+
+DEFAULT_NVIDIA_CMD = (
+    "nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,"
+    "memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits"
+)
+
+
+def _read(path):
+    try:
+        with open(path) as fh:
+            return fh.read().strip()
+    except OSError:
+        return ""
+
+
+def _pci_slot(device_dir):
+    uevent = _read(os.path.join(device_dir, "uevent"))
+    for line in uevent.splitlines():
+        if line.startswith("PCI_SLOT_NAME="):
+            return line.split("=", 1)[1]
+    return ""
+
+
+def _pci_id(device_dir):
+    uevent = _read(os.path.join(device_dir, "uevent"))
+    for line in uevent.splitlines():
+        if line.startswith("PCI_ID="):
+            return "amd:" + line.split("=", 1)[1].lower()
+    return "amd"
+
+
+def collect_amd(sys_dir):
+    for card in sorted(glob.glob(os.path.join(sys_dir, "card[0-9]*"))):
+        device = os.path.join(card, "device")
+        busy = _read(os.path.join(device, "gpu_busy_percent"))
+        if busy == "":
+            continue  # not an amdgpu compute node (e.g. a render node w/o metrics)
+        labels = {"gpu": _pci_slot(device) or os.path.basename(card),
+                  "vendor": "amd", "model": _pci_id(device)}
+        yield "gpu_utilization_percent", labels, busy
+        for name, leaf in (("gpu_memory_used_bytes", "mem_info_vram_used"),
+                           ("gpu_memory_total_bytes", "mem_info_vram_total")):
+            v = _read(os.path.join(device, leaf))
+            if v != "":
+                yield name, labels, v
+        temp = ""
+        for hwmon in sorted(glob.glob(os.path.join(device, "hwmon", "hwmon*", "temp1_input"))):
+            temp = _read(hwmon)
+            if temp != "":
+                break
+        if temp != "":
+            yield "gpu_temperature_celsius", labels, str(int(temp) / 1000.0)
+        power = _read(os.path.join(device, "power1_average"))
+        if power != "":  # empty on the R9700s -> metric absent, not zero
+            yield "gpu_power_watts", labels, str(int(power) / 1e6)
+
+
+def collect_nvidia(cmd):
+    if not cmd:
+        return
+    try:
+        out = subprocess.run(cmd, shell=True, capture_output=True, text=True,
+                             timeout=10, check=True).stdout
+    except (subprocess.SubprocessError, OSError):
+        return  # no NVIDIA GPU / no driver: tolerate, emit nothing
+    for line in out.splitlines():
+        parts = [p.strip() for p in line.split(",")]
+        if len(parts) < 7:
+            continue
+        idx, name, util, mused, mtotal, temp, power = parts[:7]
+        labels = {"gpu": idx, "vendor": "nvidia", "model": name}
+        yield "gpu_utilization_percent", labels, util
+        yield "gpu_memory_used_bytes", labels, str(int(float(mused)) * 1048576)
+        yield "gpu_memory_total_bytes", labels, str(int(float(mtotal)) * 1048576)
+        if temp not in ("", "[N/A]"):
+            yield "gpu_temperature_celsius", labels, temp
+        if power not in ("", "[N/A]"):
+            yield "gpu_power_watts", labels, power
+
+
+def collect(sys_dir, nvidia_cmd):
+    samples = {}
+    for source in (collect_amd(sys_dir), collect_nvidia(nvidia_cmd)):
+        for name, labels, value in source:
+            try:
+                value = str(float(value))  # normalise; drop trailing junk
+            except ValueError:
+                continue
+            samples.setdefault(name, []).append((labels, value))
+    return samples
+
+
+def _fmt(value):
+    f = float(value)
+    if f == int(f) and abs(f) < 1e15:
+        return str(int(f))
+    return repr(f)
+
+
+def render(samples):
+    out = []
+    for name, kind, help_text in FAMILIES:
+        out.append("# HELP %s %s" % (name, help_text))
+        out.append("# TYPE %s %s" % (name, kind))
+        for labels, value in sorted(samples.get(name, []), key=lambda x: sorted(x[0].items())):
+            lset = ",".join('%s="%s"' % (k, v) for k, v in sorted(labels.items()))
+            out.append("%s{%s} %s" % (name, lset, _fmt(value)))
+    return "\n".join(out) + "\n"
+
+
+class Handler(BaseHTTPRequestHandler):
+    sys_dir = "/sys/class/drm"
+    nvidia_cmd = DEFAULT_NVIDIA_CMD
+
+    def do_GET(self):  # noqa: N802 (http.server API)
+        if self.path.split("?")[0] not in ("/metrics", "/"):
+            self.send_error(404)
+            return
+        body = render(collect(self.sys_dir, self.nvidia_cmd)).encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "text/plain; version=0.0.4")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args):
+        pass
+
+
+def main(argv):
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--amdgpu-sysfs", default="/sys/class/drm")
+    ap.add_argument("--nvidia-cmd", default=DEFAULT_NVIDIA_CMD)
+    ap.add_argument("--serve", action="store_true", help="serve /metrics instead of one-shot")
+    ap.add_argument("--port", type=int, default=9401)
+    ap.add_argument("--bind", action="append", default=None,
+                    help="explicit bind address (repeatable); env GPU_EXPORTER_BINDS "
+                         "comma-list if unset. NEVER a wildcard.")
+    args = ap.parse_args(argv)
+
+    if not args.serve:
+        sys.stdout.write(render(collect(args.amdgpu_sysfs, args.nvidia_cmd)))
+        return 0
+
+    binds = args.bind
+    if not binds:
+        env = os.environ.get("GPU_EXPORTER_BINDS", "")
+        binds = [b.strip() for b in env.split(",") if b.strip()] or ["127.0.0.1"]
+    Handler.sys_dir = args.amdgpu_sysfs
+    Handler.nvidia_cmd = args.nvidia_cmd
+    servers = []
+    for b in binds:
+        if b in ("0.0.0.0", "::", "*"):
+            sys.stderr.write("refusing wildcard bind %s\n" % b)
+            return 2
+        servers.append(ThreadingHTTPServer((b, args.port), Handler))
+    sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
+    # EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
+    # healthy in `ss` yet times out every scrape once its SYN backlog fills —
+    # and the loopback deploy gate cannot see it. One thread per extra bind.
+    for s in servers[1:]:
+        threading.Thread(target=s.serve_forever, daemon=True).start()
+    servers[0].serve_forever()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

scripts/gpu-exporter.service

@@ -0,0 +1,35 @@
+# Systemd USER unit: gpu-exporter (both Dankclaw and Dankserver).
+#
+# Install path: ~/.config/systemd/user/gpu-exporter.service
+# Installed and enabled by deploy_gpu_exporter in scripts/deploy-remote.sh.
+#
+# Per-GPU metrics (utilization / VRAM / temperature / power) for the mixed
+# AMD + NVIDIA fleet, scraped by Prometheus. Like signal-relay it is a plain
+# user unit, not a quadlet: it runs python3 straight from the checkout (a pull
+# updates it with no copy step), a hand-written user unit can be enabled where
+# the podman v1 generator's units cannot, and with Linger=yes for dank it starts
+# at boot. It reads the host's own cgroup/drm sysfs and shells to nvidia-smi, so
+# it must run on the host — a container would need GPU/DRM passthrough for
+# nothing gained.
+#
+# BINDING. The base unit serves loopback only. deploy_gpu_exporter writes a
+# drop-in adding THIS host's LAN address (the same two-address posture
+# node-exporter and cadvisor use), so the endpoint is never a wildcard. It is
+# unauthenticated metrics on the LAN, identical to node-exporter :9100 and
+# cadvisor :8080 — Prometheus reaches same-host via host.containers.internal and
+# the Dankserver instance over the LAN from Dankclaw.
+
+[Unit]
+Description=GPU metrics exporter (amdgpu sysfs + nvidia-smi)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+# Loopback by default; the deploy drop-in appends the host LAN address.
+Environment=GPU_EXPORTER_BINDS=127.0.0.1
+ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

scripts/probe-frank-archon-kv.py

@@ -147,10 +147,21 @@ def main():
     deep = corpus(160_000)
     wide = corpus(1_100_000)
 
-    # 3.28 chars/token is the measured ratio of this corpus through the
+    # The corpus's chars/token ratio is MEASURED here, not stored. A stored 3.28
-    # froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
+    # (646,000 chars -> 196,919 tokens, 2026-10-11) had drifted to 3.20 by
-    # deploy); 98.5 % of the window leaves the response room under the ceiling.
+    # 2026-10-20 (646,299 chars -> 201,876 tokens) as the repo's docs grew, and a
-    chars = int(0.985 * maxlen * 3.28)
+    # stale constant sends leg B past --max-model-len, where the engine answers a
+    # perfectly legitimate 400 ("the request (201900 tokens) exceeds the available
+    # context size (200000)") and the leg reads as a capacity failure. One small
+    # request prices the ratio; 97 % of the window leaves the response room.
+    sample = prompt(wide, 4000, "CAL", "cal-marker")
+    cal, _, _ = post({"model": MODEL,
+                      "messages": [{"role": "user", "content": sample}],
+                      "max_tokens": 1, "temperature": 0.0,
+                      "chat_template_kwargs": {"enable_thinking": False}})
+    ratio = len(sample) / cal["usage"]["prompt_tokens"]
+    print(f"[cal] corpus ratio {ratio:.2f} chars/token")
+    chars = int(0.97 * maxlen * ratio)
     # Each marker's FIRST hyphen-separated word is unique, because show()
     # scores the hit on that word — a shared prefix would score every slot HIT
     # on any answer that says the word.
@@ -176,7 +187,7 @@ def main():
                 print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
     print(f"[B] total {time.time() - t0:.1f}s")
     # D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
-    dchars = int(0.25 * maxlen * 3.28)
+    dchars = int(0.25 * maxlen * ratio)
     dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
                for i in range(seqs)]
     print(f"[D] {seqs} concurrent streams x 512 tokens, "

scripts/serve-frank-archon.sh

@@ -15,18 +15,21 @@
 # log states every budget before anything is allocated (weights vs KV
 # split, the expert host pool's coverage of the worst case, the PLE pin,
 # the P2P answer per pair) — read it after every resize.
-# Deployed shape (2026-10-19, radiance 1.8.1): THREE decode slots x 200,000
+# Deployed shape (2026-10-20, radiance 1.8.1): FOUR decode slots x 200,000 tokens over a
-# tokens, fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in
+# >= 700,000-token fp8 KV pool, 48 GiB/rank pinned expert pool, MTP depth 2, prefix cache
-# pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab
+# in pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV
-# and the KV pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair
+# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget
-# and the budget resolver derives it from headroom, the cache bound and the host
+# resolver derives it from headroom, the cache bound and the host pool, so the slab
-# pool, so the slab size and the pool's token count are LEDGER lines: after every
+# size and the pool's token count are LEDGER lines. After every boot read them, plus
-# boot read them out of the log and check the pool still backs 3 x 200,000.
+# the two lines that size a prefill step — "activation arena: ... lends ... while
+# steps are at most N tokens" (the arena levels the expert slab borrows) and
+# "prefill staging: ... a step of more than N tokens ..." (the chunk size below which
+# pooled experts are read synchronously) — and the cache's session backing.
 #
 # The root unit (scripts/frank-archon.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
+# LimitMEMLOCK=infinity) grants the pin budget: 2 x 48 GiB expert pools + the 47.7 GiB PLE
-# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
+# table + 8 GiB prefix host tier + the checkpoint store (slots x 112 MiB) — ~159 GiB of
-# pages, which is ALSO why this script refuses to start when ttm's
+# pinned pages, which is ALSO why this script refuses to start when ttm's
 # pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
 set -euo pipefail
 

scripts/serve-frank-dragoon.sh

@@ -136,10 +136,17 @@ fi
 #      The engine re-checks size+sha256 against that catalog at boot; it cannot
 #      fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
 #      stays unset — no outbound connection at boot),
-#   2. the distro XRT's three libraries, each mounted twice (where the image
+#   2. the distro XRT's three libraries, each mounted THREE times: where the
-#      looks for XRT, and at its own path — the libraries find each other there;
+#      image looks for XRT (/opt/xilinx/xrt/lib), at their own path (the
-#      an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
+#      libraries find each other there), and at /opt/xilinx/lib/<multiarch>
-#      container, which the vendor calls out explicitly),
+#      — the NPU child searches that third prefix too, and since the host's
+#      libxrt2 was reconfigured 2026-10-20 it is the ONLY prefix the child
+#      names: omitting it crashes the lane with `No such library
+#      '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the engine
+#      supervisor takes the dead lane down on the whole container (measured
+#      2026-10-21: 88-restart boot loop). Real files only — an /opt/xilinx/xrt
+#      of symlinks into the system dir dangles inside the container, which the
+#      vendor calls out explicitly),
 #   3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
 #      hang the machine while FCLK changes speed, and the engine refuses to
 #      start when it moves — it cannot read host sysfs from inside the
@@ -190,7 +197,9 @@ if [ -n "$NPU_MODELS" ]; then
             exit 1
         fi
         REAL="$(readlink -f "$XRT_DIR/$L")"
-        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
+        MULTIARCH="${XRT_DIR##*/}"
+        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro"
+                   -v "$REAL:/opt/xilinx/lib/$MULTIARCH/$L:ro")
     done
     HELD=0
     for D in "$SYSFS"/class/drm/card*/device; do

scripts/test-changed.sh

@@ -293,6 +293,14 @@ for path in "${CHANGED[@]}"; do
             add_test tests/caddyfile.bats
             add_test tests/deployment_scoping.bats
             ;;
+        # The GPU exporter is a host process; its seam pins the collector, the
+        # user unit, the deployer function, the planner route, the Prometheus
+        # scrape jobs and the host-system GPU panels — a cross-file contract, so
+        # the whole seam rides on any of its own files.
+        scripts/gpu-exporter.py|scripts/gpu-exporter.service)
+            add_test tests/gpu_exporter.bats
+            add_test tests/deployment_scoping.bats
+            ;;
         dankclaw/udev/*)
             # Host-level udev rules are applied by manual sudo install + udevadm
             # trigger, not dispatched as a service; nothing here is bats-testable.

serverAppConfigs-docs/Agent/OMP Harness.md

@@ -52,28 +52,31 @@ Where the contract lives:
 The arithmetic (the coupled contract every endpoint's maxTokens comment
 encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
 endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
-follows is a **worked example** recomputed from the live files on 2026-10-18 —
+follows is a **worked example** recomputed from the live files on 2026-10-21 —
-a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
+a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
 maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
 
 1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
-   95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
+   95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
    bundle clamps the percentage to 1–99 and converts it to a token threshold.
 2. **Output budget** = capped at the target model's `maxTokens` — every role
    here compacts to frank-zealot, so the budget is its 16,384.
-3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
-   window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
+   dragoon session the fitting window is the ENGINE ceiling, not a slot: the
-   200,704. The registry's own comments design around an 85 % trigger (archon:
+   190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
-   170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
+   error plus the 45,056 request budget lands inside — that headroom is the
-   does hold. While `config.yml` and those comments disagree, treat 85 % as the
+   reason the ceiling moved to native on 2026-10-21. For a session compacting
-   design point and re-run this arithmetic after ANY window change — a 400 from
+   INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
-   the compaction target mid-conversation is the symptom.
+   does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
+   200,704. Treat 85 % as the design point until config.yml and the registry
+   comments agree, and re-run this arithmetic after ANY window change — a 400
+   from the compaction target mid-conversation is the symptom.
    (Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
    177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
    32,768 ≤ 200,704 zealot.)
 
 Procedure (recompute from the LIVE files every time — windows rotate with
-every swap; the block above is the 2026-10-18 state):
+every swap; the block above is the 2026-10-21 state):
 
 1. Read the `config.yml` compaction block + `models.yml` entries for source
    and target. Every endpoint's `contextWindow` comment carries its own
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
 as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
 dankclaw — the comment/config disagreed. The comments now name the real
 target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
-truth; fix comments in the same change.** A fresh instance of the same trap is
+truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
-open 2026-10-18: every `maxTokens` comment still says "compaction fires at
+disagreement is not academic: it is why long dragoon sessions kept drawing
-85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
+400s at the old 200,704 ceiling — the trigger fires late AND the est
+under-counts, and a window parked near the ceiling has no room for either.
 `maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
 input. Boot-shape OOM laddering is a different problem
 ([[Llama Cpp Serving Sizing]]).

serverAppConfigs-docs/Architecture/Stale Signal Audit.md

@@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** —
 docs, unit docstrings, and the repo's rule file (the most expensive place to
 be stale).
 
+## Three rules that outlive the incident that taught them
+
+* **A target can read down beside a healthy service.** Read `lastError` from
+  `/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the
+  protocol layer without `--metrics` (501) while `/health` answers 200 for weeks.
+* **Alert state lives where the tool says it does.** Prometheus keeps rule state in
+  process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is
+  not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a
+  host volume: silences + notification log, [[Alertmanager]]).
+* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production
+  unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape,
+  [[frank-zealot]]) beside an answering port means the metric measured a scratch boot;
+  the process shape can change between two censuses ten minutes apart.
+
 ## Order and verification
 
 1. Enumerate the surfaces (status board, metrics targets, alert rules, test

serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md

@@ -110,9 +110,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
 
 ## Recommended additions
 
-Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
+Quantized preferred. The iGPU's GTT aperture is not free RAM beside
-reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
+frank-dragoon — size every pick against *What actually fits beside
-2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
+frank-dragoon* below (the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
 all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
 `huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
 `huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
@@ -133,8 +133,34 @@ sha256 checks) or a plain `hf download` with the token in the environment —
 never on the command line.
 
 Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
-the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
+the 2.1 files already on disk), and any BF16 of #2/#4 — they double the disk
-them at GTT capacity, and they double the disk bill).
+bill, and see *What actually fits beside frank-dragoon* for why raw GTT size
+is not the real budget.
+
+## What actually fits beside frank-dragoon
+
+The 122 GB `/system_stats` figure is the GTT **aperture**, not free memory: on
+an APU the aperture and `MemAvailable` are the same physical RAM, and
+frank-dragoon pins tens of GiB of it. Measured 2026-10-21 with frank-dragoon
+live: `MemAvailable` 15.3 GiB, iGPU busy 1.75 GiB — ComfyUI itself idles at
+~10 MB, so the budget for a generation run is roughly what `free` says.
+
+Weights must land inside that budget with slack for compute buffers and
+activations (the mmproj-class lesson: price a model at ~1.5× its file set):
+
+| Pipeline (weights) | Fits beside frank-dragoon? |
+|---|---|
+| Qwen-Image 2.1 int8_convrot + w4a8 + bf16 VAE (~14.2 GB) — the two proven workflows | **No** — thrash-or-fail; do not queue it while the endpoint serves |
+| GGUF Q8_0 + w4a8 + small VAE (~13.3 GB) | No, same reason |
+| Any BF16 combination | No |
+| SDXL-Turbo / SD 1.5-class (~2–7 GB) | Yes, with real headroom |
+
+**The law: one process owns the memory — same rule as the 3090 bench law.**
+For a Qwen-Image render: stop and mask frank-dragoon
+(`sudo systemctl stop frank-dragoon` + `mask` — the mask stops
+`Restart=always` from firing mid-render), generate, unmask, verify
+`curl 192.168.0.143:7770/health`. The proven workflows above then run with
+the whole machine. Do not treat `/system_stats`'s 122 GB as an invitation.
 
 ## See Also
 

serverAppConfigs-docs/Dankclaw/ComfyUI.md

@@ -129,6 +129,12 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
 - Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
 - The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
 - The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
+- **The 122 GB `/system_stats` VRAM figure is the GTT aperture, not free
+  memory.** On the APU it is system RAM, and frank-dragoon pins most of it
+  (`MemAvailable` 15.3 GiB with the endpoint live, measured 2026-10-21). A
+  Qwen-Image-class render beside the running endpoint is thrash-or-fail; the
+  stop-mask-render-verify procedure and the fit table are in
+  [[ComfyUI Qwen-Image Workflows]] → What actually fits beside frank-dragoon.
 
 ## See Also
 

serverAppConfigs-docs/Dankserver/Plex.md

@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
 tags:
   - dankserver
   - plex
-timestamp: 2026-09-15
+timestamp: 2026-10-21
 ---
 
 # Plex
@@ -24,7 +24,7 @@ public traffic can be fronted by Caddy like every other vhost.
 | Item | Value |
 |------|-------|
 | Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
-| Image | `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (pinned; verified == the ghcr `latest` manifest digest at pin time) |
+| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
 | Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
 | Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
 | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |

serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md

@@ -81,7 +81,7 @@ Two provisioning rules, each verified live by an outage:
    appears. Create the Application with `provider_id` set (the UI does this
    automatically; raw API/ORM drives must do it explicitly).
 
-## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)
+## OAuth2 relying-party quirks (this fork; corrected live 2026-10-20)
 
 The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
 verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
@@ -100,23 +100,32 @@ implementation diverges from upstream in ways that bit us; all verified live:
 - **Provider resolution is by `client_id` query param** on the global
   `/application/o/authorize/` route; per-slug authorize paths do not exist in
   this build. Token/userinfo live at `/application/o/token/` and
-  `/application/o/userinfo/?api=1`.
+  `/application/o/userinfo/` — do **not** append `?api=1` to a Grafana API URL.
-- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
+- **NULL `authorization_flow` → authorize 500, `AttributeError: 'NoneType'
-  `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
+  object has no attribute 'slug'`.** A provider created via raw `ak shell`
-  `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
+  `create()` leaves `authorization_flow` unset; the authorize view then crashes
-  becomes the *expression source* instead of the context; every call raises a
+  building the flow planner (`flows/planner.py`) and returns "Server Error —
-  swallowed `PropertyMappingExpressionException` and the claim is skipped.
+  Powered by authentik" before any claim is built. This was previously
-  Scope-grant filtering has the same class of bug: `ScopeMapping` has no
+  mis-attributed to a broken evaluator; the crash is the missing flow. Fix: set
-  provider FK (linkage lives in the `ProviderPropertyMapping` through model)
+  `authorization_flow = default-provider-authorization-implicit-consent` (the
-  so `filter(provider=...)` always yields an empty queryset and tokens asked
+  flow every other provider in this instance already uses).
-  to carry scopes get zero. Net effect: userinfo/introspection answers with
+- **A scope mapping that returns a non-dict emits no claim.** authentik merges a
-  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
+  granted scope mapping's evaluated result into the token **as a mapping**: a
-  build until the evaluator is patched or upgraded. Clients consuming this
+  dict contributes its keys as claims; a bare string is keyed under the
-  provider must key identity off `sub`.
+  mapping's NAME, so nothing usable appears. `grafana-email-scope` returned the
-- Scope mappings for this provider exist — `grafana-email-scope` and
+  bare string `user.email`, so no `email` claim landed — the evaluator was
-  `grafana-profile-scope` are both rows in `authentik_core_propertymapping`
+  never broken (the dict-valued `grafana-profile-scope` mapping has always
-  (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
+  delivered, and the stock OpenID mappings return dicts). Fix:
-  the evaluator bug is fixed; no config change is needed then.
+  `return {"email": user.email, "email_verified": True}`. With no email, Grafana
+  v13 falls back to GET `<api_url>/emails`; authentik 404s it → "Login failed:
+  InternalError".
+- **Grafana reads identity from the standard userinfo endpoint** (`API_URL` with
+  no `?api=1`); no `USE_ID_TOKEN` is needed once the `email` claim exists.
+- **Net (2026-10-20):** the evaluator is fine; email/profile claims reach RP
+  clients. Both IdP fixes are idempotent in
+  `scripts/authentik-grafana-sso-fix.sh` (re-run after an ak-shell re-creation of
+  the provider or a blueprint import, which revert them). Grafana's half plus the
+  verified outcome (Viewer, `login = <sub hex>`, email present) are in [[Grafana]].
 
 ## LDAP provider objects (provisioned via `ak shell` ORM)
 

serverAppConfigs-docs/Infrastructure/Grafana.md

@@ -4,7 +4,7 @@ description: "Grafana visualizes the metrics that Prometheus collects from every
 tags:
   - infrastructure
   - grafana
-timestamp: 2026-09-11
+timestamp: 2026-10-20
 ---
 
 # Grafana
@@ -64,19 +64,28 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
 - Change the default admin credentials at first boot (env or UI).
 - Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
 - **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
-  no provider token (`token=None`), so no identity header ever reaches Grafana;
+  no provider token (`token=None`), so no identity header reaches Grafana; the
-  RP mode is topology-independent. The full verified contract (provider pk 9,
+  login is Grafana's own "Sign in with Authentik" button against the `grafana`
-  client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths,
+  confidential-client provider. **Verified end-to-end 2026-10-20** after four
-  Basic client auth, form-encoded token body, and the fork's userinfo-claims
+  independent breakages had made it fail. The two Authentik-side causes (a NULL
-  bug that leaves identity at the `sub` hash only) is in
+  `authorization_flow` → authorize 500; an email scope mapping that returned a
-  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana:
+  bare string so no `email` claim was emitted) and the full wire contract live in
-  The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`
+  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks"; the IdP half is
-  `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
+  re-applied by `scripts/authentik-grafana-sso-fix.sh` (idempotent — re-run after
-  users land as Viewer with `login = <sub hex>`; no email/groups/role claims
+  an ak-shell re-creation of the provider or a blueprint import, both of which
-  reach Grafana until the evaluator bug is fixed — do not write role mappings
+  revert it silently). The Grafana half is the unit's own env (`grafana.container`):
-  that depend on those claims. The local admin account stays the only Admin.
+  `SCOPES="openid email profile"` (a GitLab-style `user:email` scope crashes the
-  (The Viewer-default outcome and the `<sub hex>` login are not readable from
+  authorize view), the standard userinfo `API_URL` with **no `?api=1`**, the
-  the host without the database or an admin password. *(unverified 2026-10-18)*)
+  attribute paths `email`/`sub`/`preferred_username`, and `ALLOW_SIGN_UP=true`
+  (safe: the guard fronts the callback). Grafana v13 only GETs `<api_url>/emails`
+  when the token carries no `email` claim — so it is the email claim itself (an
+  IdP fix, above) that makes the login land, not any Grafana toggle.
+- **What a new login becomes:** a Viewer in org 1 (`ALLOW_ASSIGN_ORG=true`,
+  `ORG_ID=1`, no role-attribute mapping), with `login = <sub hex>` but the real
+  `email`. The local admin account stays the only Admin. The landing that the
+  edge-only 13-sso spec never checked is now asserted by
+  `tests/e2e/specs/20-grafana-sso.spec.ts` (RP round-trip → `/api/user` 200 with
+  an email).
 - **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
   file is a gluetun env file and it carries `HOME=/root`; systemd imports it
   for the service, podman then looks for its configuration under `/root`

serverAppConfigs-docs/Infrastructure/Observability Provisioning.md

@@ -3,7 +3,7 @@ type: reference
 title: Observability Provisioning
 description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
 tags: [observability, prometheus, grafana, provisioning, deployment]
-timestamp: 2026-10-05T00:00:00-07:00
+timestamp: 2026-10-21T00:00:00-07:00
 ---
 
 # Observability Provisioning
@@ -20,6 +20,8 @@ committed.
 | `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
 | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
 | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
+| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`) + the loopback bind |
 
 Targets and datasources reference **container names** (`grafana:3000`
 `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
 reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
 the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
 (`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
-HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
+HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
-[[Stale Signal Audit]]).
+deployed 2026-10-21; see [[Stale Signal Audit]]).
 
 ## The deploy path
 
@@ -43,8 +45,9 @@ from `deploy_observability`:
 - `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
 - `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
 - `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
-- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
+- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
-- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
+- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
 
 `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
 `alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
@@ -82,11 +85,14 @@ reports 4 alerting + 2 recording. Rules reload with the prometheus service.
 the backend (`http://prometheus:9090`), not loopback.
 
 **Add a dashboard** — drop the JSON into
-`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
+`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
-unified storage, not (only) the SQLite table — grep the API, not the DB, when
+the next `--service grafana` deploy provisions it. This is a repaired law, not
-verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
+a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
-2026-10-18: the storage-backend claim needs an admin password to confirm
+`containers` board — merged, tested, documented as "working" — had **never
-against `grafana.db` and the API side by side)*
+been provisioned**; the other boards existed on disk only because someone had
+hand-copied them once. A repo JSON is not a live dashboard until the copier
+carries it, and `deploy.sh` cannot see the gap because it only reports the
+copy it performed.
 
 ## Verify through the APIs, not the filesystem
 
@@ -95,90 +101,32 @@ against `grafana.db` and the API side by side)*
 curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
 curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
   python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
-# grafana: datasource + dashboard provisioned
+# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
-curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
+# account is the Authentik RP login). The proven surface is a real browser
+# session: warm the Anubis challenge on the vhost root, then hit
+# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
+# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
 ```
 
 The two Prometheus queries were run on 2026-10-18 and are what every number on
 this page comes from: 4 alerting + 2 recording rules, and `up` for
 `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
-`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
+`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
-listed in [[Prometheus]]. The Grafana query needs the local admin password and
+that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
-was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
+2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
-proof of installation only — see the next paragraph.
+filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
+only — see *Add a dashboard* for why a repo JSON is not a live board.
 
 Health answers from a stale process and provisioned resources can hide in
 unified storage — a file existing on disk is not proof the running server
 loaded it. See [[Stale Signal Audit]] for the never-green-target decision
 table (refused = outage, 404/HTML = config defect).
 
-## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
-
-Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
-loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
-unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
-via `host.containers.internal`) → signal-cli, addressed by number
-`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
-2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
-`curl -s http://192.168.0.143:18093/health` → `ok`, and
-`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
-lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
-`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
-structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
-the blackbox exporter. See [[Prometheus]] → Targets.
-
-**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
-`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
-the oldest sample, 2026-10-11, is already 0 — while the service itself answered
-`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
-is started with the flag, so the scrape was refused at the protocol layer (the
-flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
-stayed firing:
-the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
--u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
-on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
-metrics opt-in landed that day; the target
-reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
-How many payload generations it spanned is not readable from any host
-artifact. *(unverified 2026-10-18)*
-
-Two laws fall out of it. A target can read down beside a healthy service, so
-read `lastError` from `/api/v1/targets` before believing an outage. And rule
-state lives in Prometheus' process, not on disk — a restart re-stamps
-`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
-evidence of a fresh fault. (Alertmanager by contrast **does** persist:
-`--storage.path=/alertmanager` on a host volume keeps silences and the
-notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
-`/opt/alertmanager/data`.)
-
-Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
-metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
-([[frank-dragoon]]), which is why
-`frank-dragoon` has always been green. The active-alert count is a
-moment-in-time value and should never be documented as a constant:
-`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
-`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
-minutes later, and was empty again by 11:56Z; the relay logged the matching
-firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
-
-**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
-production `frank-zealot` root unit was stopped **on purpose** for a benchmark
-window on the 3090 — the repo Benchmarking Policy says stop the production
-service before benching — and `:7771` answered from a manually booted candidate
-arm (`systemctl is-active frank-zealot` → `failed` after
-`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
-`TimeoutStopSec`, the known shutdown shape for these engines). So
-`up{job="frank-zealot"} == 1` that day measured the trial, not the production
-shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
-and it is the repo contract.
-The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
-reinstalls and starts it, restoring boot persistence and self-healing. Never
-quote a candidate arm's flags or VRAM as fleet state — the process and its shape
-changed between two censuses ten minutes apart.
-
 ## Related
 
 - [[Observability]] — stack topology and the alert delivery chain
-- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
+  down-beside-healthy / alert-state-persistence / candidate-arm rules
 - [[Quadlet Lifecycle]] — the units that host these services
 - [[Deploy State Triage]] — how the planner selects these services

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -2,7 +2,7 @@
 type: reference
 title: "Observability on Dankclaw"
 description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
-tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
+tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
 timestamp: 2026-10-20
 ---
 
@@ -10,10 +10,10 @@ timestamp: 2026-10-20
 
 ## Summary
 
-Six containers and one host-side process: Prometheus (metrics), Grafana
+Seven containers and two host-side processes: Prometheus (metrics), Grafana
 (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
-metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
+metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
-carrying every firing group to Signal. All of it is Quadlet-managed,
+plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
 boot-persistent and loopback-only, deployed by
 `scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
 end to end; the log half is configured, healthy, and carries no data.
@@ -34,6 +34,8 @@ that proves it today.
 | Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
 | Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
 | node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
+| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
+| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
 | Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
 
 Every unit lives in `dankclaw/observability/quadlet/`, carries
@@ -44,9 +46,11 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
 the same day, and `homepage-status.timer` — the root unit that probes all of
 these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
 the probe registry `shared/homepage/services.json` to `/srv/homepage`).
-Dankserver runs one member of this family, node-exporter, as its own unit
+Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
-(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
+(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
-`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
+`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
+cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
+`gpu-exporter` (a user unit installed the same way as `signal-relay`).
 
 ## Exposure
 
@@ -63,10 +67,18 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
 in the host network namespace, and it binds exactly two addresses rather than
 the wildcard — see [[Node-Exporter]].
 
-The relay's bind is the one address a bridge container can reach on the host:
+**Same-host reachability from a bridge container is measured, not inferred**
-`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
+(2026-10-21, from inside the Prometheus container): `host.containers.internal`
-and the podman bridge subnet `10.89.0.0/16` before the DROP
+(the podman gateway) reaches host services whether they bind host-net or are
-(`scripts/deploy-remote.sh:255-262`).
+published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
+— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
+every same-host Prometheus target uses the gateway and only cross-host targets
+(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
+relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
+iptables INPUT admitting loopback, the host itself and the podman bridge subnet
+`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
+point a scrape target at a host LAN IP from its own host on the strength of
+one working example; dial the gateway and probe.
 
 ## Deployment
 
@@ -83,13 +95,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
 [[Observability Provisioning]] for what is provisioned as code and how to
 verify it through the APIs.
 
-## Current state (2026-10-20)
+## Current state (2026-10-21)
 
 * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
-  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
+  every job up — the three LLM endpoints (frank-archon on `vllm:*`,
-  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
+  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
-  and both node exporters. The four downs are structural (the `comfyui` and
+  both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
-  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
+  the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
+  downs are retired — see [[Prometheus]] → Targets.
   Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
   (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
   alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
@@ -97,30 +110,59 @@ verify it through the APIs.
 * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
   relay, and `journalctl --user -u signal-relay` records successful
   `sent to +12062476336:` lines for every rule in the set.
-* **Dashboards: working.** Provisioned datasources — Prometheus (uid
+* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
-  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
+  datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
-  installed from `shared/grafana/provisioning/`. The Loki datasource is how
+  `llm-fleet`, `host-system`, `containers`, `observability` and
-  you read logs: Loki has no web UI of its own, so Grafana Explore is the
+  `logs-overview` boards. Until 2026-10-21 the copier carried only
-  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
+  `llm-fleet.json` by name and the `containers` board had never been
-* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
+  provisioned; the deployer now globs the directory. The Loki datasource is
-  a rootless Alloy cannot read the `0640` journal files, and the container-log
+  how you read logs: Loki has no web UI of its own, so Grafana Explore is the
-  globs point at a root-storage layout that does not exist here. The fix is built
+  human surface (the guarded `loki.franksdank.cloud` vhost serves only the
-  and tested in-repo (2026-10-20): Alloy is now a **root system service on
+  API). Verify method and the copier law: [[Observability Provisioning]].
-  `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
+* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
-  Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
+  after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
-  lines and pushed 26,966 with zero drops. The **live** instance is still the old
+  per-container CPU/memory here, and on this rootless-Podman host it works on one
-  rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
+  condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
-  [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
+  namespace, so without that flag cadvisor sees only its own cgroup root and
-* **Not configured:** no blackbox exporter (the structural downs have no real
+  exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
-  uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
+  names nothing itself — the libpod socket resolved only 2 of 34 rootless
-  not yet deployed (see *Logs* above).
+  containers — but the quadlet unit name is carried in the cgroup path
+  (`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
+  `container` label. Network and filesystem are NOT exported for these containers
+  (no netns/classid attribution under the host-PID layout), so the dashboard
+  graphs only CPU, memory and OOM and does not fake the rest.
+* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
+  `deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
+  host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
+  own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
+  without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
+  `mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
+  emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
+  °C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
+  job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
+  `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
+  the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
+  node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
+* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
+  pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
+  the container-log globs pointed at a root-storage layout that does not exist
+  here) — fixed 2026-10-20 by making Alloy a **root system service on
+  `Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
+  but the stream carried only `job`/`service_name`, so every per-container
+  view was structurally empty: Loki metrics queries group only by INDEXED
+  stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
+  label (plus a `host` label) at ingestion — `loki.process` in
+  `shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
+  purpose-test law in [[Observability Provisioning]].
 
 ## Citations
 
-- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
+- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
 - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
 - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
+- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
 - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
 - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
+- `shared/blackbox/blackbox.yml` (prober modules + loopback bind)
 - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
 - Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`

serverAppConfigs-docs/Infrastructure/Prometheus.md

@@ -42,29 +42,27 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 
 The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
 
-**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
+**No job reads down structurally any more** (the three permanent-reds were
+retired 2026-10-21; census re-run after each deploy):
 
-| Job | Target | `health` | `lastError`, verbatim |
+| Job | Target | how |
-|-----|--------|----------|------------------------|
+|---|---|---|
-| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
+| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
-| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
+| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://comfyui:8188` | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service |
-| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
 
-Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
+Scraping a web root is not a health check: an HTML page fails the scrape
+protocol (`unsupported Content-Type`), which is what kept those targets red
+beside healthy services. The classification rule and the honesty law live in
+[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
+netns, config `shared/blackbox/blackbox.yml`, bound 127.0.0.1:9115; Prometheus
+reaches it at the `host.containers.internal` gateway name). Plain targets stay
+`host:port`, never URLs — the probe job's URL targets are the documented
+exception, relabeled onto the prober.
 
 **`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
 
 Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
 
-The remedy for a probe-shaped question is the **blackbox exporter**; the
-classification rule and the honesty law for it live in [[Stale Signal Audit]].
-Until the exporter is deployed, these three stay red for a structural reason:
-deleting them would
-hide a real service, and fixing them by publishing databases or enabling metrics
-would trade a wrong signal for a real exposure. Targets are `host:port`, never
-URLs.
-
 ## Volumes
 
 | Container path | Host path |

serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md

@@ -239,45 +239,92 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp
 sequence stops producing entirely and then re-prefills from token 0, which is a
 stall, not sludge.
 
-Knobs, in the order worth trying them (radiance GUIDE §4.5):
+**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are
+looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is
+the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) /
+(steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with
+decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**,
+with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only
+14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so
+a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is
+worth ~10 % of one step — not a lever.
 
-- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N
+**Four traps in the PromQL above.** Learn them before trusting a bucket:
-  steps carry decode rows only and admit nothing. Decoders keep their cadence; the
+- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e
-  prompt pays that in TTFT. Counts as `engine_steps_held_total`.
+  histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption
-- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work:
+  is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the
-  `auto` (default) reserves a row plus the draft window per decode row behind it and
+  *instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the
-  an eighth of the budget per waiting prefill/request, `off` lets the chunk take the
+  endpoint never stalls anyone".
-  whole step (admission never runs while it does), `N` pins the share. Never under
+- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is
-  the prefill stager's smallest pass — the startup line names it, and below it every
+  `decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one
-  routed layer reads its pooled experts synchronously, which costs the prefill far
+  scrape in eight — use step ≤ 30 s for a bucket table.
-  more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
+- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is
-- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own
+  one shape (`up`, or the first sample of a counter that only the current build exports) before
-  curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE
+  ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse"
-  at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of
+  turned out to belong entirely to the older one.
-  magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses
+- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query
-  expert residency to the arena.
+  above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode.
-- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every
-  shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena
-  back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing
-  at short prompts, a large TTFT gain at the far end of the window.
 
-Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
+Three instruments the PromQL above cannot reach:
-expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the
-decode rows inside that step also read experts over the link — a smaller shared budget helps twice.
-And **re-check
-`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
-pool are the same memory, and the failure looks like turn two taking as long as turn
-one.
 
-Availability is a version question, not a config question: `--busy-prefill-chunk` and
+- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
-`--prefill-interleave` are absent from older pins — prove it on the running container
+  `token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine /
-(`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`), never from the vendored
+  prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on
-tree, and check the source clone's depth before dating a flag from `git log -S`
+  the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry,
-(a `--depth 1` clone reports its own tip for every search).
+  and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call:
+  configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes.
+- **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid
+  (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so
+  cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its
+  prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high
+  prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` -
+  exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner:
+  [[frank-archon]].
+- **Bucket the arriving request too** — `histogram_quantile` over
+  `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees
+  sequences that are already running; the one waiting for its first token pays the same steps and
+  shows in no decode number.
 
-Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 =
+Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee):
-interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict
+
-for this fleet's shape are that page's.
+- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode
+  rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their
+  cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps.
+  Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone.
+- `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed
+  setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth
+  of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows
+  alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from
+  `engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill
+  stager's smallest pass — the largest arena level that lends, plus one, named on the startup line
+  `prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it
+  every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four
+  decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the
+  chunk blindly buys latency in both directions.
+- `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then
+  carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own
+  measurement says a span starts to win), while **every other step is planned at the shared
+  budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk
+  knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV
+  pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first.
+- `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the
+  arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses
+  decode — [[frank-archon]]).
+
+Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is
+reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a
+quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller
+slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the
+arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two
+taking exactly as long as turn one.
+
+Availability is a version question, not a config question: `--busy-prefill-chunk`,
+`--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running
+container (`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s
+`configuration` block, never from a vendored tree, and check a source clone's depth before dating a
+flag from `git log -S` (a `--depth 1` clone reports its own tip for every search).
+
+Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms.
 
 ## Related
 

serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md

@@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
 
 `--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
 reserves the registry price, not the actual pool. Raising the pool consumes
-real free VRAM without growing the arena; the registry over-prices the pool
+real free VRAM without growing the arena; the registry OVER-PRICES the pool
-(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
+(era G measured ~14 % under the planned B/token — the gap is the lesson, not
-**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
+the number; **never reuse a B/token ratio across geometries**), and there is a
-block-alignment loss when converting a token target to bytes). The ask lands
+~4 MB block-alignment loss to subtract when converting a token target to
-block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
+bytes. The ask lands block-aligned: verify against the engine's own
-`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
+`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot.
 
 **Driver-reported free VRAM after hours of traffic is NOT allocatable
 arena.** It is the estimator gap: the registry over-prices KV and charges
@@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
 **KV accounting — pin the pool BYTES, floor-sized:** set
 `--kv-cache-memory-bytes` PER RANK to the contract floor
 (`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
-come out of the arena/headroom and buy nothing. Measured B/token on this
+come out of the arena/headroom and buy nothing. The B/token rate is a property
-hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
+of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**;
-dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
+the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape.
-7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
+Read your live rate from the boot ledger every time; the decomposition (a
-planner's KV reserve is for ONE max-model-len request.
+dtype-independent mamba/side stream rides on top of the attention KV) is why
+the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve
+is for ONE max-model-len request.
 
 **Parallelism ceilings you cannot flag past:**
 
@@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness
 (`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
 + engine flags in the log.
 
-**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
-
-| shape | util | graphs | arena | result |
-|---|---|---|---|---|
-| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
-| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
-| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
-| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
-
-bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
-at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
-~207k per context max.
 
 ## 7. Arena audit procedure (when asked for "more experts")
 

serverAppConfigs-docs/LLM/frank-archon.md

@@ -88,25 +88,30 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
 | Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
 |---|---|
 | TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
-| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
+| Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
 | KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
 | VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
-| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
+| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
 | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
 | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
 | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
 | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
-| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
+| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
-| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
+| TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
 | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
 
-Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`):
+Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
-the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**,
+the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
-`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages
+`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
-**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the
+The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
-2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots
+sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
-(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live
+per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
-(≤ 4,194,304 px, 16,384 patches/pass).
+`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
+host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
+(= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
+(≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
+directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
+the bigger cache, paid over PCIe when those experts are hit.
 
 The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
 anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
@@ -145,6 +150,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
 
 | Window | Payload | Runtime | Shape | Replaced because | Rollback |
 |---|---|---|---|---|---|
+| 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
 | 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
 | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
 | 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
@@ -211,27 +217,44 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
   buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
   the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
-- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
+- **At 1.8.1 the split belongs to the resolver, and the source fixes its order.**
-  The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and
+  `core/mem/vram_budget.cpp` claims the card bottom-up (`capacity → already held → drift → arena →
-  `--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
+  staging → queues → prefetch → headroom = claimable`), takes the **static weights off the top as a
-  sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
+  floor, not a share**, and divides the `elastic` remainder in three lines (146-156):
-  hand. Three boots of evidence:
 
-  | shape | demand | cache given | backing |
+  ```cpp
+  exp = std::min(b.expert_floor, b.elastic);
+  kv  = std::min(kv_cap, std::max(b.elastic - exp, kv_one));
+  exp = std::min(std::max(b.expert_ceiling, b.expert_floor), b.elastic - kv);
+  ```
+
+  `kv_cap` is `KVManager::ceiling()` = `state + Σ_paged max_seqs × ceil(max_ctx / block) × bytes`
+  (`core/mem/kv.cpp:266`), so `--max-num-seqs` and `--max-model-len` set the cache's **demand**, never
+  its funding. The funding is `elastic − expert_floor`, and the floor is
+  `min(plane + spare, plane + spare + 2·unit_max − host_room)` (`core/place/planner.cpp:100-104`, with
+  `host_room = host_pool − file_stage_reserve` at `core/engine_bringup.cpp:1418-1422`). **That is the
+  cache's lever: every MiB added to `--host-pool-mib` drops the expert floor one-for-one and the cache
+  takes it** — minus `stage`, the prefill stager's ring (two layers of the experts left off the card),
+  which is charged back out of the cache whenever the cache lends (`vram_budget.cpp:193-203`). Boots of
+  evidence, all on 1.8.1:
+
+  | shape | demand | cache given | engine's `backs %` |
   |---|---|---|---|
-  | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
+  | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
   | 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
-  | 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
+  | 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
+  | 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
 
-  So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
+  The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
-  never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
+  `ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
-  slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
+  ledger line `kv: the pool backs N % … the cache can grow to … N tokens`, never arithmetic.** To buy
-  grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
+  window: raise `--host-pool-mib` (cost: pinned host RAM, and experts evicted to host cross PCIe —
-  `--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
+  `experts-from-host` was 5.99 GiB a rank at 44,928 and is 8.99 at 49,152) or cut the per-token cost
-  hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
+  (`fp8` KV). Raising slots raises demand **and** costs 10 × 111.98 MiB of pinned checkpoint-store RAM
-  era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
+  per slot. Under-sizing the pool the other way has always been a boot failure (36,864 refused a boot in
-  the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
+  the fp8 era: the host cannot hold what the cards will not). Behavioural check on the live shape:
-  `--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
+  `/health` reports `queue_cap 32`; a request beyond the slot count queues rather than being refused,
+  and the 400 class is a single request over `--max-model-len`.
 - **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
   slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
   256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
@@ -280,41 +303,104 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
   in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
   GiB, refused).
-- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
+- **Decode pays for the chunk it shares a step with, and `--busy-prefill-chunk auto` does not
-  3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
+  protect it.** Re-measured on the live 1.8.1 shape (34.8 h boot, Prometheus at 60 s,
-  1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
+  `engine_step_seconds{,_decode}_total` / `engine_steps{,_decode,_mixed,_prefill}_total` /
-  at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
+  `prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a
-  chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
+  chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's
-  decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
+  step time**, so a session decoding while another prompts advances one token per 166 ms. Bucket the
-  samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
+  harness's `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` (15 s
-  `prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
+  scrapes; per-stream rate = the gauge ÷ `vllm:num_requests_running`): median **166 t/s** under light
-  bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
+  prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×** collapse, and at 13 t/s with two
-  0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
+  sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing
-  busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
+  else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute,
-  confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
+  17.9 ms in the deep one) and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from
-  resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
+  30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second
-  [[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
+  victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s**
-  and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
+  and 15.5 % of requests wait over 10 s for their first token, with `http_queue_depth` at 2-3, while
-- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
+  median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264
-  shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
+  `vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The
-  the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
+  chunk is `min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here,
-  step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
+  and `busy_cap` is the fourth law.
-  the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
+- **`auto` is a prefill-speed dial, not a decoder-protection dial.** `Scheduler::busy_cap` reserves
-  The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
+  `1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the
-  knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
+  budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders:
-  edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
+  "not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and
-  it is trusted.
+  the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while
-- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
+  anything decodes behind it, which costs more" (`core/sched/scheduler.cpp:874-878`, 1.8.1). Live
-  adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
+  proof: `prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes
-  declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
+  with a waiter, not the minutes with a decoder. An explicit `N` is the only value that applies
-  (single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
+  "whenever the chunk shares the step".
-  today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
+- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a
-  floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
+  ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count —
-  moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
+  `max(n_tok, n_seq × (1 + spec))`, `core/engine.cpp:1142` — so a 1024-token chunk with two decode
-  the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
+  rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the
-  mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
+  prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on
-  retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
+  the boot line `prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48
-  fp8-native start.
+  routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its
+  pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The
+  vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt
+  beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**.
+- **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena
+  lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims
+  it **synchronously before the step** — units revert to their host copies (an address change, not a
+  copy) and both read streams drain — and `arena_tick` regrows the loan only **2 s after the last big
+  step** (`core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs
+  with a slab ~780 expert units smaller over a link also carrying the staged copies. `/diagnostics`
+  says the binding constraint in one line: "a prefill step streams every routed layer's pooled
+  experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
+  Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
+- **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
+  N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
+  costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
+  two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
+  ~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
+  measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
+  substitute for it.
+- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N
+  intervals, so `--prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2
+  intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span
+  starts to win), while **every other step is planned at the shared budget of 2048** — which is a
+  lending level *and* keeps the stager's ring, the one combination `--busy-prefill-chunk` cannot
+  express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's
+  cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a
+  span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's
+  second box (2×R9700, w4nl64, TP2) it was TTFT-neutral — 2.61 vs 2.56 s at 18k, 10.72 vs 10.55 at
+  75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096
+  does not boot at 3 slots; this shape is 2.
+- **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over
+  the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 %
+  replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last
+  **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`,
+  `core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about
+  `log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as
+  deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840
+  · … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every
+  `prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the
+  cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly
+  **`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs
+  retention only - the writes happen at every interval whatever the policy says (only `--prefill-span`
+  skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM
+  alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB
+  against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` -
+  the pinning room left beside the expert pool at boot - as the hard bound the ledger line names.
+  Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a
+  chain that keeps resuming at one deep point stops paying for it. Shortening
+  `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
+  `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
+  interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
+- **Slots cost KV, host RAM, rows, and pin — never speed.** `--max-num-seqs` carries no radiance
+  constraint: it adds ≈110 MiB of recurrent state per slot, 10 × 111.98 MiB of pinned checkpoint-store
+  RAM per slot (`--checkpoint-slots auto`), and declares the fused TP all-reduce over
+  `seqs × (1 + spec)` rows — `wht6` is bit-exact up to 25 rows, and the live shape is 4 × 3 = **12**, so
+  single-sequence decode stays exact and so does any step up to that many rows. Raising it also raises
+  the cache's *demand* (`ceiling = slots × window`), which the resolver funds from `elastic −
+  expert_floor` — so a slot count without matching host room is paid for by expert residency. Measured
+  2026-10-06: seqs 2 and 3 statistically tied (30.9-33.1 t/s, inside the ±3 % floor). Measured
+  2026-10-19 → 10-20 at 1.8.1: 2 → 4 slots cost **+4,224 MiB of pinned host pool** and **−3.01 GiB of
+  expert residency a card** and bought 2.76 → 5.64 GiB of cache (365,204 → 747,492 tokens). The old
+  "seqs 2 cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
+  specialization, and radiance booted seqs 2 on its first fp8-native start.
 - **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
   of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
   pinned weights, and the doubling is invisible in the weight count. Read the engine log for
@@ -396,12 +482,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
   theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
   60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
-  software theory.
+  software theory. **That mitigation is a BIOS setting and it is not in effect now:** `sudo
+  dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event
+  between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed
+  before crediting any software revert with a fix — and before believing a clean run of the arm
+  you just tested, because the confounder is still in the slot.
 - **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
   `Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
   all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
   them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
   decode, `exact` is the bit-exact alternative.
+- **The link is already at the board's ceiling — do not chase the `link-below-max` warning.**
+  Both root ports (`00:01.1`, `00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is
+  the whole budget per card and the cards' own `pcie_dpm` tables expose only x8 states.
+  `/diagnostics` still warns that the on-card switch hop (`01:00.0`, `05:00.0`) runs "x8 of its
+  x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host
+  side of the chain is x8 max, which the same endpoint reports two findings later as
+  `link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so
+  the only lever on a link-bound prefill is **less traffic over the link** (more expert
+  residency, fewer pooled experts), never a wider link. P2P resolves **on** (`auto` asks the
+  driver, the two cards reach each other's memory), so the collectives do not hairpin through host
+  RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the
+  all-reduce wins the link, the staged experts wait.
+- **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8
+  cores / 8 threads, one NUMA node, `nodes: -1` for both cards) running Jellyfin, Plex, Calibre
+  and RomM beside the engine, and `--http-threads` defaults to one thread per core. Measured
+  during a heavy prefill: load average ≈ 1.0, `jellyfin` 11 % CPU, `radiance` 7 % — **CPU
+  contention is not the current decode culprit**, so do not spend an experiment window on it;
+  re-read `uptime` and `ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks
+  different from the step-sharing one.
 - `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
   the disk tier silently never opens. The tier does survive a payload swap when arch and
   tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
@@ -445,11 +554,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 ## Contract (the wire)
 
 - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
-  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
+  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`, **`/diagnostics`**. `/health`
-  JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
+  answers 200 JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while
-  (`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
+  stopping** (`core/server/admin.cpp:385`), and the listener binds *after* the full load
-  `engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
+  (`main.cpp`: `engine.start()` then `engine.serve()`) — during the load the port refuses
-  bare `curl -fsS …/health` gate is honest and needs no body grep.
+  connections, so a bare `curl -fsS …/health` gate is honest and needs no body grep.
+- **`/diagnostics` is the instrument for "is the box, not the config, the limit"** — one call
+  returns `configuration` (every flag, so drift from the env file is checkable without the host),
+  `environment`, `derived` (the VRAM split as the ledger prints it), `hardware` (host, per-card
+  `link_chain` + `pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and `findings`
+  (level / what / why / fix). On this box it reports `experts-from-host` (5.99 GiB a rank crosses
+  PCIe when used), `link-capped` (root ports are x8 max — the board), `swap-in-use`,
+  `tp-wire-lossy`, and **`link-below-max` as a `warn` at 100 % card busy on both cards** — which is
+  structural on this board; see the x8-ceiling law under *This host, and the deploy path*.
 - **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
   only the alias; container, unit, scripts and env keep the service name). The env's contract
   line is that **radiance validates the requested model name, so every client — harness, bench
@@ -476,12 +593,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
   — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
 - **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
-  longer than the KV pool's token count is accepted then refused as too large to fit; under the
+  longer than the KV pool's token count is accepted then refused as too large to fit; a request
-  3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
+  beyond `--max-num-seqs` is held by the HTTP port (`queue_cap` = `max(8, 8 × slots)` — **32** here)
+  and answered 503 only past that, never paged or squeezed. The cache funds **3.7** full-height
+  sessions, so four conversations can be resident but a fourth near-full-height one can still
+  **preempt a running sequence from the back** to free its blocks
+  (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so its session halts and then
+  re-prefills. Meter: `vllm:num_preemptions_total` (cumulative for the boot, `/stats preemptions`);
+  `radiance:num_requests_preempted` is only the instant depth.
 - **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
-  Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
+  Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the
-  published-port quirk is same-host only). The boot ledger — the referee for every shape number
+  netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under
-  above — is in the container log (`podman logs frank-archon`)
+  `radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the
+  running/waiting gauges are `vllm:*` with no radiance-native equivalent, so a `radiance:`-only
+  query misses every latency answer. The step-phase counters are all scraped:
+  `engine_steps{,_decode,_mixed,_prefill,_held}_total`, `engine_step_seconds{,_decode}_total`,
+  `engine_prefill_chunks_capped_total`, `prefill_tokens_total`, `decode_tokens_per_second`,
+  `prefill_tokens_per_second`, `prefix_hit_cut_tokens_total`, `http_queue_depth`. Two holes: the
+  engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
+  `token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine /
+  prefill stager have no metric at all** — that half of every law above is read from the boot ledger,
+  `/stats` and `/diagnostics`, never from a graph. The boot ledger — the referee for every shape
+  number here — is in the container log (`podman logs frank-archon`);
   `/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
 
 ## Files

serverAppConfigs-docs/LLM/frank-dragoon.md

@@ -10,14 +10,15 @@ tags:
   - gfx1151
   - halogen
   - endpoint
-timestamp: 2026-10-19
+timestamp: 2026-10-21
 ---
 
 # frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
 
 Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
 under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
-(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
+(`2b12716`), ceiling raised to the native 262,144 on 2026-10-21 so the harness window is 200,000.
+This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
 
 ## Current shape
 
@@ -37,10 +38,12 @@ Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b
 unauthenticated token port 8730 stays on container loopback. Engine stdout →
 `/home/dank/logs/frank_dragoon.log`, **not** the journal.
 
-- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
+- Shape: `HALOGEN_CTX=262144` hard per-request ceiling = the checkpoint's native window (raised
-  `HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
+  2026-10-21 from 200704; `HALOGEN_ROPE_YARN` beyond native is not on plan) · `HALOGEN_KV_SLOTS=2` ·
-  enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
+  `HALOGEN_KV_POOL_POSITIONS=401408` inside the band `tests/frank_dragoon_unit.bats` enforces
-  exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
+  (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
+  exists here, the pool is the only KV lever · the pool carries TWO maximum harness-length (200k)
+  conversations at once; two literal ceiling-length requests would queue — that is a pool raise.
 - Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
   omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
   pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
@@ -53,10 +56,11 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
   (Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
   ~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
   Caddy/observability).
-- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
+- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 200000`
-  (= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
+  — sits 62,144 under the ceiling, clearing the ~45k fit-law gap (400-zone law below);
-  `reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
+  `tests/frank_dragoon_unit.bats` fails if the page's declared window and `HALOGEN_CTX` drift back
-  rides [[frank-zealot]].
+  together. `maxTokens: 45056`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`,
+  `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction rides [[frank-zealot]].
 - Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
   pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
   Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
@@ -124,14 +128,18 @@ created it — earlier topologies in .
   prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
   85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
   re-measure the warm ratio after any pool change.
-- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
+- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling, because the harness
-  155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
+  clamps `max_tokens` to `contextWindow − est − 4096`, its chars/4 `est` UNDER-counts code/agent
-  its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
+  traffic by ~3 %, and the compaction trigger fires at 95 % of the window (`config.yml
-  3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
+  thresholdPercent`, while the comments designed around 85 % — check the trigger BEFORE touching a
-  over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
+  window; that disagreement is why long sessions kept drawing 400s). The 3 % is measured, not
-  target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
+  theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 — over by 354, a 400.
-  is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
+  At the live pair (ceiling 262,144 / window 200,000) the worst-case ordinary request is
-  under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
+  200,000 − 4,096 + 3 % ≈ 201.8k, and the trigger point is 190,000 est × 1.03 + 45,056 ≈ 240.7k —
+  both inside the ceiling with ~60k of margin. Compaction rides [[frank-zealot]] at
+  ≈ threshold − keepRecent(20k) ≈ 175k measured against its 200,704-token slot. The seam in
+  `tests/frank_dragoon_unit.bats` re-runs this arithmetic at the 95 % trigger and reads the window
+  from THIS page, so ceiling/window pairs that drift back together fail the suite, not a session.
 - **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
   503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
   together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
@@ -248,9 +256,14 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
    (`dankclaw/udev/98-halogen-npu-access.rules`, installed and re-triggered by the deployer, `0666` like the GPU
    compute nodes beside it); the serve script refuses any node that is not other-accessible, so the fault cannot
    return as a crash-loop.
-2. The host's XRT reachable inside: there is **no `/opt/xilinx/xrt`** here — distro packages put the three files the
+2. The host's XRT reachable inside, at **THREE prefixes per library**: there is **no `/opt/xilinx/xrt`** here —
-   container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`) in
+   distro packages put the three files the container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`,
-   `/usr/lib/x86_64-linux-gnu`, where all three exist.
+   `libxrt_driver_xdna.so.2`) in `/usr/lib/x86_64-linux-gnu`. The serve script mounts each REAL file (never a
+   symlink) at `/opt/xilinx/xrt/lib/`, at its own path, AND at `/opt/xilinx/lib/<multiarch>/` — the NPU child
+   searches that third prefix and a missing one is fatal: since the host's `libxrt2` was reconfigured
+   2026-10-20 the child died with `No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the
+   engine supervisor took the WHOLE container down on it (88-restart loop, chat dead beside a healthy GPU line).
+   A dead lane is not survivable here — arm the lane only with all three prefixes mounted.
 3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
    while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
    engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**

shared/alloy/config.alloy

@@ -17,6 +17,9 @@ logging {
 // Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
 // host network, so it pushes by loopback, never by container name — a root
 // container is not on the rootless dankclaw_default network.
+// GRAFANA_HOST_LABEL is replaced with this host's name by
+// copy_alloy_config at deploy time; it must not appear verbatim in the
+// installed config.
 loki.write "default" {
   endpoint {
     url = "http://127.0.0.1:3100/loki/api/v1/push"
@@ -32,7 +35,34 @@ loki.write "default" {
 loki.source.journal "journal" {
   format_as_json = true
   labels = {
-    job = "systemd-journal",
+    job  = "systemd-journal",
+    host = "GRAFANA_HOST_LABEL",
+  }
+  forward_to = [loki.process.relabel.receiver]
+}
+
+// Promote the fields that ARE the point of the log stream to real Loki labels.
+// Without this the journal arrives as one opaque JSON line per entry and every
+// per-container view is structurally impossible: Loki metrics queries can only
+// group by INDEXED stream labels, so `sum by (container_name)` returns a single
+// anonymous series and the logs-overview board read empty beside 6.5k
+// lines/5min of live traffic (measured 2026-10-21). CONTAINER_NAME is what
+// podman's journald logging writes (uppercase); host units lack it and stay on
+// the host-only stream. `format_as_json = true` above guarantees every line
+// reaching this stage is a JSON object, so there is no non-JSON branch to
+// protect — alloy 1.21's json stage has no `on_error` attribute (validated
+// against the pinned image); a malformed line would drop, which is the
+// correct fate for a line Loki could not index anyway.
+loki.process "relabel" {
+  stage.json {
+    expressions = {
+      container = "CONTAINER_NAME",
+    }
+  }
+  stage.labels {
+    values = {
+      container = "container",
+    }
   }
   forward_to = [loki.write.default.receiver]
 }

shared/blackbox/blackbox.yml

@@ -0,0 +1,18 @@
+# Blackbox exporter modules — the honest answer to "is this HTTP service
+# answering?". A Prometheus scrape cannot express it (an HTML front page is a
+# 200 that fails the scrape protocol), which is why the old `services` and
+# `comfyui` jobs sat permanently red on `unsupported Content-Type`.
+# See serverAppConfigs-docs/Architecture/Stale Signal Audit.md §1.
+# Loopback bind: the image's default 0.0.0.0:9115 would ride the LAN on this
+# Network=host container. Prometheus (bridge) still reaches it — its traffic
+# arrives on the podman gateway interface, which is not loopback.
+network:
+  listen_address: 127.0.0.1:9115
+
+modules:
+  http_2xx:
+    prober: http
+    timeout: 10s
+    http:
+      preferred_ip_protocol: ip4
+      valid_status_codes: [] # default: 2xx + 3xx

shared/grafana/provisioning/dashboards/containers.json

@@ -0,0 +1,137 @@
+{
+  "uid": "containers",
+  "title": "Containers — Dankclaw & Dankserver",
+  "description": "Per-container CPU and memory for the quadlet app stacks, from cadvisor (a rootless quadlet on each host). cadvisor keys each series by the cgroup path, and Prometheus relabels the quadlet unit name (app.slice/<name>.service) into a `container` label. Panels use only the metric families cadvisor actually exports for these rootless containers: CPU usage and memory working-set/limit/OOM. Network and filesystem are NOT exported for them (no netns/classid attribution under the rootless+host-PID layout), so no panels pretend otherwise. See serverAppConfigs-docs/Infrastructure/Observability.md.",
+  "tags": ["fleet", "cadvisor", "containers"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "host",
+        "label": "Host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\"}, host)", "refId": "host_var" },
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      },
+      {
+        "name": "container",
+        "label": "Container",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\"}, container)", "refId": "container_var" },
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "row",
+      "title": "CPU",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+      "panels": []
+    },
+    {
+      "id": 2,
+      "type": "timeseries",
+      "title": "CPU usage (cores)",
+      "description": "CPU-seconds/s per container = cores busy. A steady high line is a busy service; a spike is a burst (a render, an index, a backup). Only containers are shown, not the host or its daemons.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 1 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "sum by (host, container) (rate(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m]))",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "bargauge",
+      "title": "Top containers by CPU (cores)",
+      "description": "Instant ranking over the selected window. Tells you which container to look at when the host is loaded.",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 1 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "max": null, "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 4 } ] } }, "overrides": [] },
+      "options": { "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false }, "displayMode": "gradient", "orientation": "horizontal", "showUnfilled": true },
+      "targets": [
+        {
+          "expr": "topk(12, sum by (host, container) (rate(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m])))",
+          "legendFormat": "{{host}} / {{container}}",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "row",
+      "title": "Memory",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 10 },
+      "panels": []
+    },
+    {
+      "id": 5,
+      "type": "timeseries",
+      "title": "Memory working set",
+      "description": "container_memory_working_set_bytes per container — the number the OOM killer acts on. Cache-inclusive and reclaimable; a flat high line is resident data, a sawtooth is churn.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 11 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "timeseries",
+      "title": "Memory vs host total (%)",
+      "description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "type": "timeseries",
+      "title": "OOM kills",
+      "description": "container_oom_events_total rate. Any non-zero line means the kernel killed a process inside that container — the honest signal that its memory limit is too tight for its real working set.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 20, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "sum by (host, container) (rate(container_oom_events_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m]))",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/host-system.json

@@ -0,0 +1,204 @@
+{
+  "uid": "host-system",
+  "title": "Hosts \u2014 Dankclaw & Dankserver",
+  "description": "node_exporter for both servers. Starter set \u2014 throw-at-the-wall breadth; refine the windows and drop the noise. The picker selects the host; CPU/load/memory/filesystem/network are per-node.",
+  "tags": ["host", "system"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "node",
+        "label": "host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(up{job=~\"node-.*\"}, job)", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      },
+      {
+        "name": "gpuhost",
+        "label": "gpu host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(gpu_utilization_percent, host)", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1, "type": "stat", "title": "Uptime",
+      "description": "Time since boot. A drop means a reboot.",
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [{ "expr": "time() - node_boot_time_seconds{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "stat", "title": "Memory used",
+      "description": "1 - MemAvailable/MemTotal. The honest \"in use\" figure (excludes reclaimable cache).",
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.8 }, { "color": "red", "value": 0.92 }] } }, "overrides": [] },
+      "targets": [{ "expr": "1 - node_memory_MemAvailable_bytes{job=~\"$node\"} / node_memory_MemTotal_bytes{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "stat", "title": "Load 5m / core",
+      "description": "Per-core load5. >1 sustained means runnable tasks are queuing behind cores.",
+      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "stat", "title": "Open file handles",
+      "description": "FDs allocated vs the kernel maximum. Rising toward the max precedes EMFILE across the host.",
+      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 0.9 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_filefd_allocated{job=~\"$node\"} / node_filefd_maximum{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "timeseries", "title": "CPU busy %",
+      "description": "CPU seconds/s in each mode = cores busy per mode; the stack sums to total cores in use (16/host here). A user or system spike points at what is loading the box; steal > 0 means the hypervisor is throttling.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 10, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,mode) (rate(node_cpu_seconds_total{job=~\"$node\",mode!=\"idle\",mode!=\"iowait\"}[5m]))", "legendFormat": "{{job}} {{mode}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 6, "type": "timeseries", "title": "Load average",
+      "description": "load1/5/15 per host.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_load1{job=~\"$node\"}", "legendFormat": "{{job}} load1", "refId": "A" },
+        { "expr": "node_load5{job=~\"$node\"}", "legendFormat": "{{job}} load5", "refId": "B" },
+        { "expr": "node_load15{job=~\"$node\"}", "legendFormat": "{{job}} load15", "refId": "C" }
+      ]
+    },
+    {
+      "id": 7, "type": "timeseries", "title": "Memory breakdown",
+      "description": "Total, free, buffers+cache. Reclaimable cache is not pressure; MemAvailable is the real headroom.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_memory_MemTotal_bytes{job=~\"$node\"} - node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} used", "refId": "A" },
+        { "expr": "node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} available", "refId": "B" }
+      ]
+    },
+    {
+      "id": 8, "type": "timeseries", "title": "Filesystem free %",
+      "description": "Per mountpoint. The media/library stacks fill disks.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 0.15 }] } }, "overrides": [] },
+      "targets": [
+        { "expr": "node_filesystem_avail_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"} / node_filesystem_size_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"}", "legendFormat": "{{job}} {{mountpoint}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 9, "type": "timeseries", "title": "Disk throughput",
+      "description": "Read/write bytes/s per device.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_disk_read_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} R", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_disk_written_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} W", "refId": "B" }
+      ]
+    },
+    {
+      "id": 10, "type": "timeseries", "title": "Disk busy %",
+      "description": "Fraction of time the device had I/O in flight. Near 1 = the device is the bottleneck.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] },
+      "targets": [
+        { "expr": "rate(node_disk_io_time_seconds_total{job=~\"$node\"}[5m])", "legendFormat": "{{job}} {{device}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 11, "type": "timeseries", "title": "Network throughput",
+      "description": "Receive (up) / transmit (down) bytes/s per interface.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} RX", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_network_transmit_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} TX", "refId": "B" }
+      ]
+    },
+    {
+      "id": 12, "type": "timeseries", "title": "Network errors / drops",
+      "description": "Interface errors and drops \u2014 the signature of a saturated link or a bad NIC/cable.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} RX err", "refId": "A" },
+        { "expr": "sum by (job,device) (rate(node_network_transmit_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} TX err", "refId": "B" }
+      ]
+    },
+    {
+      "id": 13, "type": "timeseries", "title": "GPU busy %",
+      "description": "Per-GPU utilization from gpu-exporter (amdgpu gpu_busy_percent / nvidia-smi). The picker above selects the host; each series is one card.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 36 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_utilization_percent{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 14, "type": "timeseries", "title": "GPU VRAM used",
+      "description": "Device memory in use. On the R9700s (frank-archon) and the 3090 (frank-zealot) this is the loaded model's footprint.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 36 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_memory_used_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 15, "type": "timeseries", "title": "GPU VRAM used %",
+      "description": "Used / total device memory. Near 100% is the pre-OOM wall.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 44 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
+      "targets": [
+        { "expr": "100 * gpu_memory_used_bytes{host=~\"$gpuhost\"} / gpu_memory_total_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 16, "type": "timeseries", "title": "GPU temperature",
+      "description": "Package temperature. AMD edge temp (hwmon temp1_input); NVIDIA gpu temp.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 44 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "celsius", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_temperature_celsius{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 17, "type": "timeseries", "title": "GPU power draw",
+      "description": "Board power draw. Emitted only where readable: NVIDIA and the Dankclaw iGPU. The R9700s have no sysfs power sensor, so they are simply absent here — never a fabricated 0.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 52 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "watt", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_power_watts{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/llm-fleet.json

@@ -194,7 +194,7 @@
       "id": 6,
       "type": "timeseries",
       "title": "Firing fleet alerts",
-      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
+      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
       "gridPos": {
         "h": 8,
         "w": 12,

shared/grafana/provisioning/dashboards/logs-overview.json

@@ -0,0 +1,243 @@
+{
+  "uid": "logs-overview",
+  "title": "Logs \u2014 Loki (systemd journal)",
+  "description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
+  "tags": [
+    "logs",
+    "loki"
+  ],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "templating": {
+    "list": [
+      {
+        "name": "container",
+        "label": "container",
+        "type": "query",
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "query": {
+          "query": "label_values({job=\"systemd-journal\"}, container)",
+          "refId": "LokiVariableQueryEditor-VariableQuery"
+        },
+        "includeAll": true,
+        "multi": true,
+        "refresh": 2,
+        "current": {
+          "text": "All",
+          "value": "$__all"
+        }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "timeseries",
+      "title": "Log volume by container (lines/s)",
+      "description": "Per-container journal line rate. The `container` label is promoted at ingestion by Alloy (loki.process) from podman's CONTAINER_NAME journald field - Loki metrics can only group by indexed stream labels.",
+      "gridPos": {
+        "h": 9,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "custom": {
+            "fillOpacity": 20,
+            "stacking": {
+              "mode": "normal"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 2,
+      "type": "timeseries",
+      "title": "Total log rate",
+      "description": "All journal lines per second.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))",
+          "legendFormat": "all lines/s",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "timeseries",
+      "title": "Error-ish line rate (text match)",
+      "description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "color": {
+            "mode": "fixed",
+            "fixedColor": "red"
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))",
+          "legendFormat": "error lines/s",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "timeseries",
+      "title": "Errors by container",
+      "description": "Which unit is throwing. Rank by the top offenders.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} |~ \"(?i)(error|panic|fatal)\" [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 5,
+      "type": "stat",
+      "title": "Error lines (last 15m)",
+      "description": "Absolute count over the window, not a rate.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 50
+              },
+              {
+                "color": "red",
+                "value": 500
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))",
+          "instant": true,
+          "legendFormat": "errors/15m",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "logs",
+      "title": "Journal (filtered)",
+      "description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
+      "gridPos": {
+        "h": 12,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "options": {
+        "showTime": true,
+        "wrapLogMessage": true,
+        "sortOrder": "Descending",
+        "enableLogDetails": true
+      },
+      "targets": [
+        {
+          "expr": "{job=\"systemd-journal\", container=~\"$container\"}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/observability.json

@@ -0,0 +1,385 @@
+{
+  "uid": "observability",
+  "title": "Observability \u2014 self-monitoring",
+  "description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
+  "tags": [
+    "observability",
+    "meta"
+  ],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "stat",
+      "title": "Targets up",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 1)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 2,
+      "type": "stat",
+      "title": "Targets down",
+      "description": "Every scrape target that is down. ZERO IS THE HEALTHY STATE: the HTML-probe jobs and the disabled Gitea metrics were retired 2026-10-21 (HTTP liveness now rides the blackbox exporter, job \"services\"). A non-zero count here is a real regression - read /api/v1/targets lastError before triaging.",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 4,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 0)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "stat",
+      "title": "TSDB head series",
+      "description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 8,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_tsdb_head_series",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "stat",
+      "title": "Discovered targets",
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 13,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(prometheus_sd_discovered_targets)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 5,
+      "type": "stat",
+      "title": "Prometheus mem",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "process_resident_memory_bytes{job=\"prometheus\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "stat",
+      "title": "Grafana up",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            {
+              "type": "value",
+              "options": {
+                "1": {
+                  "text": "UP",
+                  "color": "green"
+                },
+                "0": {
+                  "text": "DOWN",
+                  "color": "red"
+                }
+              }
+            }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "up{job=\"grafana\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "type": "table",
+      "title": "Target health",
+      "description": "Every scrape target and its current up. Sort by Value to surface regressions.",
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "options": {
+        "sortBy": [
+          {
+            "displayName": "Value",
+            "sortOrder": "Asc"
+          }
+        ]
+      },
+      "targets": [
+        {
+          "expr": "up",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 8,
+      "type": "timeseries",
+      "title": "Scrape interval drift (p99 actual)",
+      "description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 9,
+      "type": "timeseries",
+      "title": "Scrape samples/s",
+      "description": "Total samples ingested across all jobs.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(rate(scrape_samples_scraped[5m]))",
+          "legendFormat": "samples/s",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 10,
+      "type": "timeseries",
+      "title": "Loki request rate",
+      "description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))",
+          "legendFormat": "{{route}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 11,
+      "type": "timeseries",
+      "title": "Exporter process CPU",
+      "description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 21
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))",
+          "legendFormat": "{{job}}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}

shared/prometheus/prometheus.yml

@@ -60,20 +60,31 @@ scrape_configs:
       - targets: ["loki:3100"]
     metrics_path: "/metrics"
 
-  # ComfyUI (no native /metrics; probe the UI)
+  # HTTP liveness, probed honestly through the blackbox exporter. Scraping a
-  - job_name: "comfyui"
+  # web root is NOT a health check — an HTML front page fails the scrape
-    metrics_path: "/"
+  # protocol (`unsupported Content-Type`), which is exactly why the old
-    static_configs:
+  # `services` and `comfyui` jobs were red forever beside healthy services
-      - targets: ["comfyui:8188"]
+  # (Stale Signal Audit §1). The prober is a host-netns container reached at
-
+  # the podman gateway name (HostAddr= on the prometheus unit); it in turn
-  # Blackbox-style probes for service up/down (published LAN ports;
+  # reaches both the host LAN IP and bridge container names. comfyui's UI probe
-  # gitea and comfyui are covered by their own jobs above)
+  # lives here now — its own metrics-scrape job is retired, one owner per
+  # service.
   - job_name: "services"
-    metrics_path: "/"
+    metrics_path: /probe
+    params:
+      module: [http_2xx]
     static_configs:
       - targets:
-          - "192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
+          - "http://192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
-          - "192.168.0.143:3005"   # OpenWebUI
+          - "http://192.168.0.143:3005"   # OpenWebUI
+          - "http://comfyui:8188"         # ComfyUI UI (gfx1151 generator)
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: "host.containers.internal:9115"
 
   # LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
   # this container, not the guarded vhosts: these are machine clients and
@@ -131,3 +142,88 @@ scrape_configs:
       - targets: ["192.168.0.235:7770"]
         labels:
           service: frank-archon
+
+  # Per-container metrics from cadvisor (a rootless quadlet on each host).
+  # cadvisor keys every series by the cgroup path; for a quadlet container that
+  # path is …/app.slice/<name>.service[/libpod-payload-<hex>], so the unit NAME
+  # lives in it. The relabel below pulls that name out into a `container` label
+  # and keeps only the per-unit aggregate — cadvisor's raw host root,
+  # system.slice daemons and machine.slice scopes carry no such name and are
+  # dropped, as is the per-CPU-core split (only cpu="total" survives), so the
+  # scrape stays small. cadvisor cannot bind per-interface, so it is reached at
+  # the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
+  # DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
+  # connection whether the port is a host-net bind or a published port (the
+  # original note here claimed published ports were exempt; measured 2026-10-21
+  # from inside this container: LAN refused, gateway answers). Cross-host
+  # targets (dankserver) have no hairpin problem and stay plain LAN.
+  - job_name: "cadvisor-dankclaw"
+    static_configs:
+      - targets: ["host.containers.internal:8080"]
+        labels:
+          host: dankclaw
+    metric_relabel_configs:
+      - source_labels: [id]
+        regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
+        target_label: container
+        replacement: '$1'
+      # Keep only the unit-level cgroup; drop its payload/control children so a
+      # container is not counted twice.
+      - source_labels: [id]
+        regex: '.*/\.service/.*'
+        action: drop
+      # Drop everything that is not a named unit (no `container` was extracted).
+      - source_labels: [container]
+        regex: '.+'
+        action: keep
+      # cpu="total" for cpu series; metrics with no cpu label (memory, network,
+      # filesystem) are kept.
+      - source_labels: [cpu]
+        regex: '^$|total'
+        action: keep
+      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
+        action: labeldrop
+
+  - job_name: "cadvisor-dankserver"
+    static_configs:
+      - targets: ["192.168.0.235:8080"]
+        labels:
+          host: dankserver
+    metric_relabel_configs:
+      - source_labels: [id]
+        regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
+        target_label: container
+        replacement: '$1'
+      - source_labels: [id]
+        regex: '.*/\.service/.*'
+        action: drop
+      - source_labels: [container]
+        regex: '.+'
+        action: keep
+      - source_labels: [cpu]
+        regex: '^$|total'
+        action: keep
+      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
+        action: labeldrop
+
+  # Per-GPU metrics (gpu_utilization_percent / gpu_memory_used_bytes /
+  # gpu_memory_total_bytes / gpu_temperature_celsius / gpu_power_watts) from the
+  # host-process gpu-exporter (scripts/gpu-exporter.py). One job per host, same
+  # reach rules as node/cadvisor: Dankclaw is scraped at the podman gateway
+  # because the netavark hairpin quirk makes Prometheus fail to dial its own
+  # host LAN IP; Dankserver is plain LAN. The gpu label (PCI BDF on AMD, index on
+  # NVIDIA) + vendor/model come from the exporter, so the GPU row on the
+  # host-system dashboard filters by host. Power is emitted only where readable
+  # (NVIDIA + the Dankclaw iGPU); the R9700s have no sysfs power, so they simply
+  # have no gpu_power_watts series — no fabricated zero.
+  - job_name: "gpu-dankclaw"
+    static_configs:
+      - targets: ["host.containers.internal:9401"]
+        labels:
+          host: dankclaw
+
+  - job_name: "gpu-dankserver"
+    static_configs:
+      - targets: ["192.168.0.235:9401"]
+        labels:
+          host: dankserver

tests/caddyfile.bats

@@ -193,7 +193,18 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
 }
 
 @test "prometheus targets are host:port (no URL schemes)" {
-    ! grep -nE '^\s*-\s*"?http' "$REPO_ROOT/shared/prometheus/prometheus.yml"
+    # Plain scrape targets are host:port; a URL target is only meaningful for
+    # the blackbox prober job, whose relabel_configs copy it to __param_target
+    # and point __address__ at the prober. Anywhere else a URL target is the
+    # old HTML-probe defect wearing a new hat.
+    awk '
+        /job_name:/ { job=$0; sub(/.*"/, "", job); sub(/".*/, "", job); probe = 0 }
+        /metrics_path: \/probe/ { probe = 1 }
+        /^[[:space:]]*-[[:space:]]*"http/ && !probe {
+            print FILENAME ":" FNR ": URL target outside a prober job: " $0; bad = 1
+        }
+        END { exit bad }
+    ' "$REPO_ROOT/shared/prometheus/prometheus.yml"
 }
 
 @test "observability containers run as container root (rootless = host user)" {

tests/deployment_flow.bats

@@ -125,3 +125,28 @@ load helpers
     [ ! -e "$sandbox/frank-default-sightsound" ]
     [ -e "$BATS_TEST_TMPDIR/.AGENTTRASH/$(date +%Y%m%d)-retired-deploy-state/frank-default-sightsound" ]
 }
+
+@test "the client and remote allowlists agree per target" {
+    # Two gates guard `deploy.sh --service X`: the client's valid_service and
+    # the remote runner's service_is_allowed. When only one side gained a
+    # service the other answered "is not deployed on <target>" and the only
+    # repair path was a fleet-wide drain — cadvisor and gpu-exporter both fell
+    # through this crack within a day of each other. The lists MUST agree;
+    # searxng-network is the one sanctioned exception (client-only network
+    # unit, no remote dispatch handler).
+    run env OUT="$BATS_TEST_TMPDIR" bash -c '
+        grep -oE "dank(claw|server):[a-z0-9-]+" \
+            <(sed -n "/^valid_service()/,/^}/p" "$1/scripts/deploy.sh") \
+            | grep -v "dankclaw:searxng-network" | sort -u > "$OUT/cli"
+        grep -oE "dank(claw|server):[a-z0-9-]+" \
+            <(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
+            | sort -u > "$OUT/rem"
+        diff "$OUT/cli" "$OUT/rem"
+    ' _ "$REPO_ROOT"
+    [ "$status" -eq 0 ] || { echo "deploy.sh vs deploy-remote.sh allowlist drift:"; echo "$output"; return 1; }
+    # Drift aside, the gate must actually know the host-process exporters —
+    # their first-ever deploy has to pass the client gate.
+    for pair in dankclaw:cadvisor dankclaw:gpu-exporter dankserver:cadvisor dankserver:gpu-exporter; do
+        grep -q "$pair" "$BATS_TEST_TMPDIR/cli" || { echo "client gate missing $pair"; return 1; }
+    done
+}

tests/e2e/playwright.config.ts

@@ -29,8 +29,14 @@ export default defineConfig({
       // green suite with a test that does not exist. 17- joined on 2026-09-16
       // (the hermes vhost, whose upstream is a host address rather than a
       // container name, so it needs its own pass condition). 18- joined on
-      // 2026-09-24 (OpenWebUI SSO through authentik).
+      // 2026-09-24 (OpenWebUI SSO through authentik). 20- joined on 2026-10-21
-      testMatch: /(10|12|13|16|17|18)-.*\.spec\.ts/,
+      // (Grafana's OAuth2 relying-party login, which the edge-only 13- never
+      // follows). 21- joined 2026-10-21 (the open-notebook vhost, deployed
+      // 2026-10-19 - the newest guarded app, in neither GUARDED list nor any
+      // spec until now: the 17-hermes "uncovered vhost broke silently" class).
+      // 22- joins Prometheus's app-is-really-there proof (its own API), closing
+      // the "not a login wall but maybe a 404" hole 20- closed for Grafana.
+      testMatch: /(10|12|13|16|17|18|20|21|22)-.*\.spec\.ts/,
       use: { storageState: ".auth/qa-e2e.json" },
       dependencies: ["setup"],
     },

tests/e2e/specs/14-guard.spec.ts

@@ -15,7 +15,7 @@ const BASE = "https://www.franksdank.cloud";
 // Every vhost that Caddy routes through anubis:8923 (see dankclaw/Caddyfile).
 // Keep this list in lockstep with the Caddyfile: a new guarded vhost must be
 // added here or the guard-chain audit is incomplete.
-const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki"];
+const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki", "notebook"];
 
 // Markers that prove app content leaked past the guard to an anonymous user.
 const LEAK_MARKERS: Record<string, string[]> = {
@@ -25,6 +25,7 @@ const LEAK_MARKERS: Record<string, string[]> = {
   "seerr.franksdank.cloud": ["Request Movie"],
   "grafana.franksdank.cloud": ["Dashboards"],
   "prometheus.franksdank.cloud": ["Expression Browser", "Graph"],
+  "notebook.franksdank.cloud": ["Open Notebook"],
 };
 
 test("homepage root (/) is guarded for unauthenticated users", async ({ page }) => {

tests/e2e/specs/20-grafana-sso.spec.ts

@@ -0,0 +1,56 @@
+import { expect, test } from "@playwright/test";
+
+// Grafana (grafana.franksdank.cloud) is an OAuth2 RELYING PARTY, not an
+// outpost-fronted proxy: the guard chain (13-sso) only proves an authenticated
+// session reaches Grafana's login page. The login itself happens through
+// Grafana's own "Sign in with Authentik" button, which the edge test never
+// follows — that is exactly how the RP login could be dead (500 on authorize,
+// or an email-claim gap bouncing back to /login) while 13- still passed green.
+// This spec drives the real RP round-trip and asserts a live Grafana session.
+
+const HOST = "grafana.franksdank.cloud";
+const CHALLENGE = "Making sure you're not a bot";
+
+test("SSO: Grafana relying-party login lands a real session", async ({ page, request }) => {
+  test.setTimeout(180_000);
+
+  // Warm the guard: the vhost runs its own Anubis proof-of-work on first visit.
+  await page.goto(`https://${HOST}/`);
+  await expect
+    .poll(
+      async () => !(await page.locator("body").innerText()).includes(CHALLENGE),
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+
+  // The "Sign in with Authentik" button targets this RP start URL exactly.
+  // Going straight there is the same code path without depending on the SPA's
+  // button markup.
+  await page.goto(`https://${HOST}/login/generic_oauth`);
+
+  // The chain: grafana -> auth (implicit consent, the session is already
+  // there) -> back to grafana with the code -> grafana provisions the user and
+  // redirects to "/". A dead provider/claims config bounces back to /login.
+  await expect
+    .poll(
+      async () => {
+        const u = new URL(page.url());
+        return u.hostname === HOST && u.pathname !== "/login";
+      },
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+
+  const body = await page.locator("body").innerText();
+  expect(body).not.toContain("Login failed");
+  expect(body.toLowerCase()).not.toContain("powered by authentik"); // IdP flow shell
+  expect(body).not.toContain(CHALLENGE);
+
+  // A genuine logged-in session exists: /api/user answers 200 with an email,
+  // not 401. This is the assertion the edge-only spec could never make.
+  const resp = await page.request.get(`https://${HOST}/api/user`);
+  expect(resp.status()).toBe(200);
+  const user = await resp.json();
+  expect(typeof user.email).toBe("string");
+  expect(user.email.length).toBeGreaterThan(0);
+});

tests/e2e/specs/21-notebook.spec.ts

@@ -0,0 +1,52 @@
+import { expect, test } from "@playwright/test";
+
+// open-notebook (notebook.franksdank.cloud), deployed 2026-10-19, is a guarded
+// vhost whose outpost (notebook-proxy) fronts an app with its OWN password
+// login - the same shape as hermes (17-) and OpenWebUI (18-): the guard chain
+// (Anubis PoW -> authentik outpost) must hand off to the app, and the app's own
+// login page is the expected landing. Until now this host appeared in neither
+// the 13-/14- GUARDED lists nor any dedicated spec, so a broken upstream would
+// have stayed invisible - the exact 17-hermes "uncovered vhost" failure class.
+// 14- now covers the unauthenticated gate; this spec owns authenticated
+// reachability with a POSITIVE app marker, not just "not a login wall".
+
+const VHOST = "https://notebook.franksdank.cloud/";
+const CHALLENGE = "Making sure you're not a bot";
+
+test("notebook vhost reaches the open-notebook app through the guard chain", async ({ page }) => {
+  test.setTimeout(180_000);
+  await page.goto(VHOST);
+
+  // The Anubis proof-of-work clears in a real browser and the outpost hands off
+  // to the app; the user must never be parked on an authentik flow.
+  await expect
+    .poll(async () => !(await page.locator("body").innerText()).includes(CHALLENGE), { timeout: 90_000 })
+    .toBe(true);
+  await expect
+    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+    .toBe("notebook.franksdank.cloud");
+  expect(
+    new URL(page.url()).pathname.startsWith("/if/"),
+    "must not be parked on an authentik flow",
+  ).toBe(false);
+  expect(
+    await page.locator('input[name="uidField"]').count(),
+    "must not show the authentik login form",
+  ).toBe(0);
+
+  // Positive identity: the page really is open-notebook's own login, not a
+  // Caddy error or the IdP shell. The app answering its own login is the pass.
+  await expect(page, "destination title must be Open Notebook").toHaveTitle(/open notebook/i, {
+    timeout: 30_000,
+  });
+  await expect(page.locator("body"), "open-notebook login marker").toContainText(
+    /enter your password/i,
+    { timeout: 30_000 },
+  );
+  const body = await page.locator("body").innerText();
+  expect(body, "must not leak the IdP flow shell").not.toContain("Powered by authentik");
+  expect(body, "must not be a Caddy upstream error").not.toContain("Bad Gateway");
+
+  const resp = await page.request.get(VHOST);
+  expect(resp.status(), "open-notebook must answer, not error").toBeLessThan(500);
+});

tests/e2e/specs/22-prometheus-sso.spec.ts

@@ -0,0 +1,33 @@
+import { expect, test } from "@playwright/test";
+
+// Prometheus (prometheus.franksdank.cloud) sits behind the same embedded outpost
+// as Grafana. 13-sso and 11-admin only ever assert it is "not a login wall and
+// under 500" - which a Prometheus serving a 404 or a blank page would ALSO pass.
+// This is the app-is-really-there proof 20- provides for Grafana: with the
+// standard session the outpost hands off silently to the Prometheus UI, and the
+// app's own API answers success. A guard-route or upstream regression that lets
+// any non-Prometheus answer through fails here.
+
+test("SSO: Prometheus app answers its own API through authentik", async ({ page, request }) => {
+  test.setTimeout(180_000);
+  await page.goto("https://prometheus.franksdank.cloud/");
+
+  // Challenge clears, outpost hands off silently to the app (no credential prompt).
+  await expect
+    .poll(
+      async () => !(await page.locator("body").innerText()).includes("Making sure you're not a bot"),
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+  await expect
+    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+    .toBe("prometheus.franksdank.cloud");
+  expect(await page.locator('input[name="uidField"]').count(), "no authentik form").toBe(0);
+  await expect(page, "must land on the Prometheus UI").toHaveTitle(/prometheus/i, { timeout: 30_000 });
+
+  // Load-bearing assertion: the Prometheus API itself answers, authenticated,
+  // through the outpost. An IdP bounce, a 404, or a raw upstream error fails here.
+  const api = await request.get("https://prometheus.franksdank.cloud/api/v1/status/buildinfo");
+  expect(api.status(), "Prometheus API must answer 200 through the guard").toBe(200);
+  expect((await api.json()).status, "Prometheus API must report success").toBe("success");
+});

tests/frank_archon_unit.bats

@@ -145,7 +145,7 @@ nosudo_bin() {
     # (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
     # seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
     # tied inside the +-3% noise floor — slots buy concurrency, not speed.
-    grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_MAX_SEQS=4$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
@@ -158,7 +158,7 @@ nosudo_bin() {
     # so the pool's token count is a boot-ledger line — verified live by the
     # capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
     grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
-    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=49152$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
     grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
     # Prefix cache: pinned host RAM only (the disk tier was removed), and the
@@ -177,8 +177,11 @@ nosudo_bin() {
     [ "$ttmt" -gt "$ttmm" ]
     pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
     ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_CACHE_MIB=/{print $2}' "$ENV_FILE")
-    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
+    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host + MNBT + the auto
-    ledger=$(( 2 * pool + 48824 + ph + 2048 ))
+    # checkpoint store (MAX_SEQS x 10 slots of 112 MiB -- the store is pinned host
+    # memory too, and it grows with the slot count).
+    seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
+    ledger=$(( 2 * pool + 48824 + ph + 2048 + seqs * 10 * 112 ))
     [ "$ttmm" -ge "$ledger" ]
     # The .rad size floor is positive (a truncated convert must fail fast).
     grep -q '^FRANK_ARCHON_RAD_MIN_MIB=' "$ENV_FILE"

tests/frank_dragoon_unit.bats

@@ -3,7 +3,7 @@
 # of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
 # Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
 # digest-pinned), driven by the root unit's serve script — 2 slots over ONE
-# 200,704-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0)
+# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
 # and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
 # shape, which itself cut over the halogen 27B-dense shape on 2026-10-06.
 #
@@ -85,7 +85,7 @@ good_env() {
         "FRANK_DRAGOON_VISION_BYTES=0"
         "FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
         "HALOGEN_MODEL_ID=frank-dragoon"
-        "HALOGEN_CTX=200704"
+        "HALOGEN_CTX=262144"
         "HALOGEN_KV_POOL_POSITIONS=401408"
         "HALOGEN_KV_SLOTS=2"
         "HALOGEN_SPEC_ADAPT=0"
@@ -159,9 +159,11 @@ good_env() {
     reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
     # The pool is ONE budget behind the slots (boot ledger form: "2 slots over
     # ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
-    # a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx
+    # a per-slot private pool. Since 2026-10-21 the ceiling is the checkpoint's
-    # (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be
+    # native 262,144 while the pool stays 401,408 (raised 2026-10-18): two
-    # resident at once — frank-zealot parity on total in-flight context. What
+    # MAXIMUM harness-length (200,000-token) conversations are resident at once;
+    # two literal ceiling-length requests would queue — widening that is a pool
+    # raise, not a window move.
     # still binds: the pool must back ONE full-length request (below that the
     # advertised per-request ceiling would be unreachable, hence pool >= ctx),
     # it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
@@ -287,7 +289,7 @@ $(cat "$TEST_TMP/serve.log")"
     [[ "$output" == *"--name frank-dragoon-halogen"* ]]
     [[ "$output" == *"--publish 7770:8731"* ]]
     [[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
-    [[ "$output" == *"-e HALOGEN_CTX=200704"* ]]
+    [[ "$output" == *"-e HALOGEN_CTX=262144"* ]]
     [[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
     [[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
     [[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
@@ -306,27 +308,32 @@ $(cat "$TEST_TMP/serve.log")"
 
 @test "the shipped env keeps the harness fit law inside the ceiling" {
     # HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400
-    # naming the limit, never a silent truncation). The law is INVARIANT to
+    # naming the limit, never a silent truncation). The law: the harness
-    # the value: the harness window must sit ~45k UNDER it, because the
+    # window must sit ~45k UNDER the ceiling, because the harness clamps
-    # harness clamps max_tokens = min(45056, window − est − 4096), compacts
+    # max_tokens = min(45056, window − est − 4096), compacts at the trigger
-    # at 85 % of its declared window, and its chars/4 est UNDER-counts the
+    # percent of its declared window, and its chars/4 est UNDER-counts the
-    # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens at
+    # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens
-    # est 152,938 on 2026-10-07 — with window == ceiling, every request over
+    # at est 152,938 on 2026-10-07 — with window ≈ ceiling, every long
-    # ~65 % of the window was a guaranteed 400 before compaction could fire).
+    # request is a guaranteed 400 before compaction can fire). The 2026-10-21
-    # The shipped pair is ctx 200704 / window 155648 (lowered from
+    # pair: ceiling 262144 (native) under a harness window of 200,000.
-    # 262144/217088 on 2026-10-17); the gap must absorb the error at BOTH
+    # The declared window is read from the endpoint page — the one owner —
-    # monotone maxima of est + budget (est ≤ window − 4096):
+    # so env, models.yml and page that disagree FAIL HERE, not as a 400
-    #   full budget at the compaction threshold: 1.03*132300 + 45056
+    # mid-session (the recurring failure this seam exists for).
-    #   clamped budget at the clamp's zero point: 1.03*151552 + 0
+    grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE"
-    # and the compaction request that rides on frank-zealot (its
+    ctx="$(sed -nE 's/^HALOGEN_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
-    # 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k).
+    page="$ROOT/serverAppConfigs-docs/LLM/frank-dragoon.md"
-    grep -qE '^HALOGEN_CTX=200704$' "$ENV_FILE"
+    grep -qE 'contextWindow: 200000' "$page"
-    ctx=200704
+    hw="$(grep -oE 'contextWindow: [0-9]+' "$page" | head -1 | grep -oE '[0-9]+')"
-    hw=155648; mt=45056
+    mt=45056
-    thr=$(( hw * 85 / 100 ))
+    # Invariant checked at the live 95 % trigger (config.yml
+    # thresholdPercent) — strictly stronger than the 85 % design point.
+    # Both monotone maxima of est + budget (est ≤ window − 4096):
+    thr=$(( hw * 95 / 100 ))
     [ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
     zero=$(( hw - 4096 ))
     [ $(( zero * 103 / 100 )) -le "$ctx" ]
+    # The compaction request rides on frank-zealot's 200,704-token slot:
+    # input ≈ threshold − keepRecent(20k).
     [ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
 }
 
@@ -423,9 +430,12 @@ npu_fixture() {
     # the stock root:render 0660 mode, and the serve gate checks exactly that.
     touch "$TEST_TMP/accel0"
     chmod 666 "$TEST_TMP/accel0"
-    mkdir -p "$TEST_TMP/xrt"
+    # basename carries a multiarch shape: the third mount prefix is derived
+    # from the XRT dir's basename (the NPU child searches
+    # /opt/xilinx/lib/<multiarch>/, measured 2026-10-21).
+    mkdir -p "$TEST_TMP/x86_64-linux-gnu"
     for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
-        touch "$TEST_TMP/xrt/$l"
+        touch "$TEST_TMP/x86_64-linux-gnu/$l"
     done
     mkdir -p "$TEST_TMP/npu/qwen3-embedding-0.6b/devices"
     mkdir -p "$TEST_TMP/sys/class/drm/card1/device"
@@ -435,7 +445,7 @@ npu_fixture() {
         "HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"
         "FRANK_DRAGOON_NPU_DIR=$TEST_TMP/npu"
         "FRANK_DRAGOON_DEV_ACCEL=$TEST_TMP/accel0"
-        "FRANK_DRAGOON_XRT_DIR=$TEST_TMP/xrt"
+        "FRANK_DRAGOON_XRT_DIR=$TEST_TMP/x86_64-linux-gnu"
         "FRANK_DRAGOON_SYSFS=$TEST_TMP/sys"
     )
 }
@@ -465,11 +475,15 @@ $(cat "$TEST_TMP/serve.log")"
     ! grep -q 'accel0' "$TEST_TMP/serve.log"
 }
 
-@test "the lane arms the accel node, the staged model dir and each XRT library twice" {
+@test "the lane arms the accel node, the staged model dir and each XRT library three times" {
-    # Each library mounts at the image's XRT path AND at its own path: the
+    # Each library mounts at the image's XRT path, at its own path (the distro
-    # distro libraries find each other where they were installed, and an
+    # libraries find each other where they were installed), and at
-    # /opt/xilinx/xrt of symlinks into the system dir dangles inside the
+    # /opt/xilinx/lib/<multiarch>/ — the prefix the NPU child searches and
-    # container (the vendor calls that out by name).
+    # names when it is missing (`No such library
+    # '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'`, measured
+    # 2026-10-21: the lane died and the supervisor took the whole container
+    # with it, 88-restart loop). An /opt/xilinx/xrt of symlinks into the
+    # system dir dangles inside the container (the vendor calls that out).
     podman_stub
     good_env
     npu_fixture
@@ -484,8 +498,9 @@ $(cat "$TEST_TMP/serve.log")"
     [[ "$output" != *":/models/npu"* ]]
     [[ "$output" == *"-e HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"* ]]
     for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
-        [[ "$output" == *"-v $TEST_TMP/xrt/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
-        [[ "$output" == *"-v $TEST_TMP/xrt/$l:$TEST_TMP/xrt/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:$TEST_TMP/x86_64-linux-gnu/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/lib/x86_64-linux-gnu/$l:ro"* ]]
     done
     # The Flash payload still rides the same container: the lane is additive.
     [[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
@@ -560,7 +575,7 @@ $(cat "$TEST_TMP/serve.log")"
     podman_stub
     good_env
     npu_fixture
-    rm "$TEST_TMP/xrt/libxrt_driver_xdna.so.2"
+    rm "$TEST_TMP/x86_64-linux-gnu/libxrt_driver_xdna.so.2"
     run_with_log
     [ "$status" -ne 0 ]
     [[ "$output" == *"libxrt_driver_xdna.so.2 missing"* ]]

tests/gpu_exporter.bats

@@ -0,0 +1,227 @@
+#!/usr/bin/env bats
+# Seam: the GPU metrics exporter's black-box contract.
+#
+# The exporter reads amdgpu sysfs (one card per /sys/class/drm/cardN that has a
+# device/gpu_busy_percent file) and, on the box that has a working NVIDIA stack,
+# one `nvidia-smi --query-gpu` CSV, and emits the unified `gpu_*` Prometheus
+# families the host-system dashboard graphs. Both sources are injectable so the
+# whole thing is testable with fixtures on a GPU-less dev box.
+#
+# Only the inputs a consumer sees are pinned: the metric families, their label
+# set, and that absent hardware yields ABSENT series (never a fabricated 0 — a
+# wrong number on a dashboard gets believed). The R9700s expose no sysfs
+# power1_average, so those cards legitimately carry no gpu_power_watts line.
+
+load helpers
+
+EXPORTER="$REPO_ROOT/scripts/gpu-exporter.py"
+
+# Build a fixture amdgpu sysfs tree. $BATS_TEST_TMPDIR/sys
+mk_amd() { # gpu busy vram_used vram_total power_uw_or_blank temp_mc
+    local card=$1 busy=$2 used=$3 total=$4 power=$5 temp=$6
+    local d="$SYS/card$card/device"
+    mkdir -p "$d/hwmon/hwmon9"
+    printf '%s\n' "$busy" > "$d/gpu_busy_percent"
+    printf '%s\n' "$used" > "$d/mem_info_vram_used"
+    printf '%s\n' "$total" > "$d/mem_info_vram_total"
+    printf '%s\n' "$power" > "$d/power1_average"          # blank => absent on R9700
+    [ -n "$temp" ] && printf '%s\n' "$temp" > "$d/hwmon/hwmon9/temp1_input"
+    printf 'DRIVER=amdgpu\nPCI_ID=1002:7551\nPCI_SLOT_NAME=0000:03:00.0\n' > "$d/uevent"
+}
+
+setup() {
+    SYS="$BATS_TEST_TMPDIR/sys"
+    mkdir -p "$SYS"
+}
+
+@test "exporter exists and compiles" {
+    [ -f "$EXPORTER" ]
+    python3 -m py_compile "$EXPORTER"
+}
+
+@test "amd card with full sysfs yields util/vram/temp/power with the right labels and units" {
+    mk_amd 1 42 1073741824 34359738368 250000000 71000   # 250 W, 71 C
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    # util is a raw percent, VRAM verbatim bytes, temp milli->C, power microW->W.
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="amd"[^}]*\} 42$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*\} 1073741824$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_memory_total_bytes\{[^}]*\} 34359738368$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_temperature_celsius\{[^}]*\} 71$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_power_watts\{[^}]*\} 250$' || { echo "$output"; return 1; }
+    # identity labels: the PCI slot is a stable unique id, vendor marks the family.
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{.*gpu="0000:03:00\.0"' || { echo "$output"; return 1; }
+}
+
+@test "card with an empty power file and no hwmon yields util/vram but omits power and temp" {
+    mk_amd 1 99 34108338176 34208743424 "" ""           # the R9700 shape
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*\} 99$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*\} 34108338176$'
+    # Absent means ABSENT, not a zero the dashboard would plot as a real reading.
+    ! echo "$output" | grep -Eq '^gpu_power_watts\{' || { echo "fabricated power line: $output"; return 1; }
+    ! echo "$output" | grep -Eq '^gpu_temperature_celsius\{' || { echo "fabricated temp line: $output"; return 1; }
+}
+
+@test "nvidia csv yields util/vram/temp/power tagged vendor=nvidia" {
+    nvidia="$BATS_TEST_TMPDIR/nvidia-smi"
+    cat > "$nvidia" <<'STUB'
+echo '0, NVIDIA GeForce RTX 3090, 93, 23004, 24576, 76, 397.40'
+STUB
+    chmod +x "$nvidia"
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd "$nvidia"
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="nvidia"[^}]*\} 93$' || { echo "$output"; return 1; }
+    # memory.used MiB -> bytes (23004 * 1048576)
+    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*vendor="nvidia"[^}]*\} 24121442304$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_temperature_celsius\{[^}]*vendor="nvidia"[^}]*\} 76$'
+    echo "$output" | grep -Eq 'gpu_power_watts\{[^}]*vendor="nvidia"[^}]*\} 397\.4$'
+    echo "$output" | grep -Eq 'model="NVIDIA GeForce RTX 3090"'
+}
+
+@test "a failing nvidia command is tolerated: amd still emits and exit is 0" {
+    mk_amd 1 50 1000000 2000000 "" 40000
+    failing="$BATS_TEST_TMPDIR/nvidia-fail"
+    printf '#!/bin/sh\nexit 1\n' > "$failing"; chmod +x "$failing"
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd "$failing"
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="amd"[^}]*\} 50$'
+    ! echo "$output" | grep -q 'vendor="nvidia"'
+}
+
+@test "no GPUs at all yields only HELP/TYPE headers and exits 0" {
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    # families are declared even when empty, so the dashboard/queries stay valid.
+    echo "$output" | grep -q '# TYPE gpu_utilization_percent gauge'
+    echo "$output" | grep -q '# TYPE gpu_memory_used_bytes gauge'
+    ! echo "$output" | grep -Eq '^gpu_utilization_percent\{'
+}
+
+@test "output is valid Prometheus text" {
+    mk_amd 1 42 1000 2000 "" 50000
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ]
+    # every non-comment line is "<name>{labels} value" or "<name> value"
+    printf '%s\n' "$output" | python3 -c '
+import sys, re
+pat = re.compile(r"^[a-z_]+(\{[^}]*\})? [-+0-9.eE+]+$")
+for i, l in enumerate(sys.stdin):
+    l = l.rstrip("\n")
+    if not l or l.startswith("#"):
+        continue
+    assert pat.match(l), "malformed line %d: %r" % (i, l)
+print("prometheus text ok")
+'
+}
+
+# ---- deployment + wiring contracts -------------------------------------
+# The exporter is a HOST-PROCESS user unit (not a quadlet): it must read the
+# host's own DRM sysfs and shell to nvidia-smi, which a container could not do
+# without GPU passthrough. These tests pin the cross-file wiring a reader
+# cannot eyeball — a lone change on one side leaves the metric silently absent.
+
+@test "the base unit runs the exporter from the checkout and never binds a wildcard" {
+    local u="$REPO_ROOT/scripts/gpu-exporter.service"
+    [ -f "$u" ]
+    grep -q '^ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve$' "$u" \
+        || { echo "unit does not run the exporter --serve from the checkout"; return 1; }
+    grep -q '^Restart=always$' "$u"
+    grep -q '^WantedBy=default.target$' "$u"
+    # Base unit binds loopback only; the per-host LAN address is added by a
+    # deploy drop-in, so a checked-in unit is never a wildcard.
+    grep -q '^Environment=GPU_EXPORTER_BINDS=127.0.0.1$' "$u"
+    ! grep -qE '0\.0\.0\.0' "$u"
+}
+
+@test "the deployer installs, drops in the LAN bind, and health-gates gpu-exporter" {
+    local handler="$REPO_ROOT/scripts/deploy-remote.sh"
+    grep -q '^deploy_gpu_exporter() {' "$handler"
+    # Per-host LAN from $TARGET (never a hardcoded host, never a wildcard).
+    grep -q 'dankclaw) lan=192.168.0.143' "$handler"
+    grep -q 'dankserver) lan=192.168.0.235' "$handler"
+    # Drop-in binds loopback + this host's LAN as one comma list.
+    grep -q "GPU_EXPORTER_BINDS=127.0.0.1,%s" "$handler"
+    grep -q 'systemctl --user enable gpu-exporter.service' "$handler"
+    grep -q 'systemctl --user restart gpu-exporter.service' "$handler"
+    # Health gate waits on the loopback metrics endpoint.
+    grep -q 'http://127.0.0.1:9401/metrics' "$handler"
+    # Dispatched from the central case.
+    grep -q 'gpu-exporter) deploy_gpu_exporter ;;' "$handler"
+    # Allowlisted on BOTH hosts.
+    grep -q 'dankclaw:gpu-exporter' "$handler"
+    grep -q 'dankserver:gpu-exporter' "$handler"
+}
+
+@test "the planner selects gpu-exporter for its files on each target" {
+    local t f
+    for t in dankclaw dankserver; do
+        for f in scripts/gpu-exporter.py scripts/gpu-exporter.service; do
+            run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target "$t" --paths "$f"
+            [ "$status" -eq 0 ]
+            [ "$output" = "gpu-exporter" ] || { echo "$t/$f -> '$output'"; return 1; }
+        done
+    done
+}
+
+@test "Prometheus scrapes gpu-exporter where node/cadvisor are reachable on each host" {
+    # Cross-file contract, same posture as node-exporter: the local instance is
+    # reached via the podman gateway (this Prometheus is a bridge container and
+    # a host-process bind is refused at the host LAN IP); the Dankserver
+    # instance is a different machine, so plain LAN routing reaches it. If either
+    # address drifts from the exporter's bind, GPU panels go blank with no error.
+    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
+    grep -q 'job_name: "gpu-dankclaw"' "$cfg" || { echo "no gpu-dankclaw job"; return 1; }
+    grep -q 'job_name: "gpu-dankserver"' "$cfg" || { echo "no gpu-dankserver job"; return 1; }
+    grep -q '"host.containers.internal:9401"' "$cfg" \
+        || { echo "gpu-dankclaw does not scrape the podman gateway"; return 1; }
+    grep -q '"192.168.0.235:9401"' "$cfg" \
+        || { echo "gpu-dankserver does not scrape the LAN address"; return 1; }
+    # A vhost target would be red forever (Prometheus cannot solve Anubis).
+    ! grep -qE '"[^"]*franksdank[^"]*:9401"' "$cfg"
+}
+
+@test "the host-system dashboard graphs the gpu_* families the exporter emits" {
+    local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/host-system.json"
+    [ -f "$dash" ]
+    python3 -c "
+import json
+d = json.load(open('$dash'))
+exprs = [t['expr'] for p in d['panels'] for t in p.get('targets', [])]
+for want in ('gpu_utilization_percent', 'gpu_memory_used_bytes',
+             'gpu_temperature_celsius', 'gpu_power_watts'):
+    assert any(want in e for e in exprs), ('missing', want, exprs)
+print('gpu dashboard panels ok')
+"
+}
+
+@test "--serve answers on EVERY declared bind, not just the first" {
+    # The deployed unit serves loopback + the host LAN address, and Prometheus
+    # reaches the host through the gateway — which dials the LAN-class bind.
+    # A server that only runs serve_forever() on the first socket leaves the
+    # second one listen()ing-but-never-accepting: the deploy health gate on
+    # loopback passes, every scrape on the other bind dies on the SYN backlog,
+    # and no gate but this one sees it. (This exact bug shipped 2026-10-21 and
+    # left both gpu-* targets down beside a green deploy.)
+    mk_amd 0 42 1073741824 34359738368 "" 71000
+    port=$((19400 + RANDOM % 500))
+    python3 "$EXPORTER" --serve --amdgpu-sysfs "$SYS" --nvidia-cmd false \
+        --bind 127.0.0.1 --bind 127.0.0.2 --port "$port" \
+        > "$BATS_TEST_TMPDIR/serve.log" 2>&1 &
+    local pid=$!
+    trap 'kill "$pid" 2>/dev/null' EXIT
+    local ok=0
+    for _ in $(seq 1 20); do
+        curl -fsS -m2 "http://127.0.0.1:$port/metrics" >/dev/null 2>&1 && { ok=1; break; }
+        sleep 0.25
+    done
+    [ "$ok" -eq 1 ] || { kill "$pid" 2>/dev/null; return 1; }
+    # Both binds must ANSWER within 2 s each — the second one is the bug.
+    for b in 127.0.0.1 127.0.0.2; do
+        curl -fsS -m2 "http://$b:$port/metrics" | grep -q '^gpu_utilization_percent{' \
+            || { echo "bind $b did not answer /metrics"; kill "$pid" 2>/dev/null; return 1; }
+    done
+    kill "$pid" 2>/dev/null
+    trap - EXIT
+}

tests/observability_quadlet.bats

@@ -21,7 +21,7 @@ load helpers
 
 UNIT_DIR="$REPO_ROOT/dankclaw/observability/quadlet"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
-UNITS=(prometheus grafana loki alloy node-exporter alertmanager)
+UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager blackbox-exporter)
 
 # container_section <unit> — the [Container] key/value block only
 container_section() {
@@ -33,7 +33,7 @@ service_section() {
     awk '/^\[Service\]/{f=1;next} /^\[/{f=0} f' "$1"
 }
 
-@test "all six observability units exist under dankclaw/observability/quadlet/" {
+@test "all eight observability units exist under dankclaw/observability/quadlet/" {
     local svc
     for svc in "${UNITS[@]}"; do
         [ -f "$UNIT_DIR/$svc.container" ] || { echo "missing $svc.container"; return 1; }
@@ -277,6 +277,7 @@ loki http://127.0.0.1:3100/ready
 alloy http://127.0.0.1:12345/-/healthy
 node-exporter http://127.0.0.1:9100/metrics
 alertmanager http://127.0.0.1:9093/-/healthy
+cadvisor http://127.0.0.1:8080/healthz
 PAIRS
 }
 
@@ -568,3 +569,188 @@ assert any('ALERTS' in e for e in exprs), exprs
     [ "$status" -eq 0 ]
     [ "$output" = "alertmanager" ]
 }
+
+@test "cadvisor exports container metrics only because of --cgroupns=host, and binds exactly its two consumers" {
+    # Podman starts every container in a private cgroup namespace, so a cadvisor
+    # without --cgroupns=host sees only its own cgroup root and exports a single
+    # series (measured 2026-10-20: 1 series before the flag, 996 after). That
+    # flag and the host /sys mount are the entire mechanism — pin them, plus the
+    # publish set: loopback (host probe + health gate) and this host's LAN
+    # address (Prometheus reaches a host-published port there). Never a wildcard.
+    local unit lan
+    for pair in "dankclaw/observability/quadlet:192.168.0.143" "dankserver/observability/quadlet:192.168.0.235"; do
+        unit="$REPO_ROOT/${pair%%:*}/cadvisor.container"
+        lan="${pair##*:}"
+        [ -f "$unit" ] || { echo "missing $unit"; return 1; }
+        grep -q -- '--cgroupns=host' "$unit" || { echo "$unit lacks --cgroupns=host"; return 1; }
+        grep -q -- '--pid=host' "$unit" || { echo "$unit lacks --pid=host"; return 1; }
+        grep -q '^Volume=/sys:/sys:ro$' "$unit" || { echo "$unit lacks the host /sys mount"; return 1; }
+        grep -q '^WantedBy=default.target$' "$unit" || { echo "$unit is not boot-persistent"; return 1; }
+        grep -q '^Restart=always$' "$unit" || { echo "$unit is not self-healing"; return 1; }
+        grep -q '^HealthCmd=' "$unit" || { echo "$unit has no health check"; return 1; }
+        grep -q "^PublishPort=127.0.0.1:8080:8080\$" "$unit" || { echo "$unit must publish loopback :8080"; return 1; }
+        grep -q "^PublishPort=$lan:8080:8080\$" "$unit" || { echo "$unit must publish $lan:8080"; return 1; }
+        ! grep -qE '^PublishPort=(0\.0\.0\.0|:)' "$unit" || { echo "$unit publishes on a wildcard"; return 1; }
+    done
+}
+
+@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
+    # Cross-file contract, measured 2026-10-21 from inside the Prometheus
+    # container: a bridge container cannot dial its OWN host's LAN IP — the
+    # hairpin refuses published ports exactly like host-net binds, so
+    # cadvisor-dankclaw must scrape via the podman gateway (the original pin
+    # here demanded the LAN address and shipped a permanently-down target).
+    # dankserver is a different machine: plain LAN there. And the whole
+    # dashboard depends on the id -> container relabel: cadvisor names nothing
+    # itself, the quadlet unit name is only in the cgroup path. If either half
+    # changes alone, the Containers dashboard is blank.
+    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
+    grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
+    grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
+    grep -q '"host.containers.internal:8080"' "$cfg" \
+        || { echo "cadvisor-dankclaw does not scrape via the podman gateway"; return 1; }
+    ! grep -q '"192.168.0.143:8080"' "$cfg" \
+        || { echo "cadvisor-dankclaw dials its own host LAN IP — hairpin refuses it"; return 1; }
+    grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
+    # The relabel that turns …/app.slice/<name>.service… into a `container`.
+    grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }
+    grep -q 'target_label: container' "$cfg" || { echo "relabel does not set the container label"; return 1; }
+}
+
+@test "the Containers dashboard is provisioned from git" {
+    local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/containers.json"
+    [ -f "$dash" ]
+    python3 -c "
+import json
+d = json.load(open('$dash'))
+assert d['uid'] == 'containers', d['uid']
+names = [t['expr'] for p in d['panels'] for t in p.get('targets', [])]
+# Only families cadvisor actually exports for rootless quadlet containers are
+# graphed; network/fs are absent for them, so a network/fs panel would be a
+# permanent empty lie.
+assert any('container_cpu_usage_seconds_total' in e for e in names), names
+assert any('container_memory_working_set_bytes' in e for e in names), names
+assert not any('container_network' in e or 'container_fs_' in e for e in names), names
+"
+}
+
+@test "the planner selects cadvisor for its unit on each target" {
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
+        --paths dankclaw/observability/quadlet/cadvisor.container
+    [ "$status" -eq 0 ]
+    [ "$output" = "cadvisor" ]
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver \
+        --paths dankserver/observability/quadlet/cadvisor.container
+    [ "$status" -eq 0 ]
+    [ "$output" = "cadvisor" ]
+}
+
+@test "the grafana copier provisions EVERY dashboard json, not a hand-listed subset" {
+    # Behavioral, not a text grep: the copier is extracted and executed with a
+    # stubbed run_privileged against a sandbox. The fleet provider turns each
+    # file in its directory into a dashboard, so a file the copier skips is a
+    # dashboard that silently never exists (the containers board lived in the
+    # repo, never reached /opt, and its fixes were undeployable).
+    # BATS_TEST_TMPDIR: per-test, removed by bats itself; a mktemp dir with a
+    # RETURN trap races bats' own trap machinery and vanishes mid-function.
+    local sandbox="$BATS_TEST_TMPDIR/sandbox"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_grafana_provisioning()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        export GRAFANA_PROV_ROOT="$sandbox/opt/grafana"
+        copy_grafana_provisioning
+    )
+    local f
+    for f in "$REPO_ROOT"/shared/grafana/provisioning/dashboards/*.json; do
+        [ -f "$sandbox/opt/grafana/provisioning/dashboards/$(basename "$f")" ]
+    done
+    [ -f "$sandbox/opt/grafana/provisioning/dashboards/provider.yml" ]
+    [ -f "$sandbox/opt/grafana/provisioning/datasources/prometheus.yml" ]
+}
+
+@test "changes under shared/blackbox/ plan the prober" {
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
+        --paths shared/blackbox/blackbox.yml
+    [ "$status" -eq 0 ]
+    [ "$output" = "blackbox-exporter" ]
+}
+
+@test "the copier runs the same code path the deployer runs" {
+    # Textually pinned so a refactor cannot quietly route the deployer around
+    # the copy_* functions the seams execute.
+    grep -q '        prometheus)' "$REMOTE_DEPLOY"
+    grep -q 'copy_prometheus_config' "$REMOTE_DEPLOY"
+    grep -q 'copy_alertmanager_config' "$REMOTE_DEPLOY"
+    grep -q 'copy_loki_config' "$REMOTE_DEPLOY"
+    grep -q '        blackbox-exporter)' "$REMOTE_DEPLOY"
+    grep -q 'copy_blackbox_config' "$REMOTE_DEPLOY"
+}
+
+@test "the alloy copier installs the host-labeled config, token-free" {
+    # Executes copy_alloy_config for real against a sandbox (stubbed
+    # run_privileged). A surviving GRAFANA_HOST_LABEL token would ship a
+    # config whose host label is a literal placeholder — the exact class of
+    # silent-label defect this tranche retired the board for.
+    local sandbox="$BATS_TEST_TMPDIR/opt"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        # NOTE: VAR=x func does not bind for shell functions — export.
+        export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
+        copy_alloy_config
+    )
+    [ -f "$sandbox/config/config.alloy" ]
+    grep -q 'host = "dankclaw",' "$sandbox/config/config.alloy"
+    ! grep -q "GRAFANA_HOST_LABEL" "$sandbox/config/config.alloy"
+}
+
+@test "the blackbox copier installs the prober config" {
+    local sandbox="$BATS_TEST_TMPDIR/blackbox"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_blackbox_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        export BLACKBOX_CONF_ROOT="$sandbox"
+        copy_blackbox_config
+    )
+    [ -f "$sandbox/blackbox.yml" ]
+    grep -q "listen_address: 127.0.0.1:9115" "$sandbox/blackbox.yml"
+}
+
+@test "the alloy copier refuses to install a config the engine rejects" {
+    # The syntax gate: alloy attributes are position-sensitive and a
+    # misplaced one only explodes at run time (crash-loop behind a green
+    # deploy). Prove the copier refuses a config the pinned image won't
+    # validate, and installs nothing. Needs the alloy image; skipped
+    # where it is absent (the copier's gate is likewise image-gated).
+    if ! podman image exists localhost/franksdank/alloy:latest; then
+        skip "franksdank/alloy image not present on this box"
+    fi
+    local sandbox="$BATS_TEST_TMPDIR/opt" cfg="$BATS_TEST_TMPDIR/broken.alloy"
+    sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
+    printf 'bogus_attr = "x"\n' >> "$cfg"
+    ! (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        cp() { if [ "$1" = "$cfg" ]; then return 1; fi; command cp "$@"; }
+        export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
+        copy_alloy_config
+    )
+    [ ! -f "$sandbox/config/config.alloy" ]
+}
+
+@test "the installed alloy config is engine-valid" {
+    # Runs the pinned image's own validator over the shipped config. The
+    # 2026-10-21 defect (`on_error` on the labels stage, which alloy 1.21
+    # does not define) passed every textual seam and only failed at boot.
+    if ! podman image exists localhost/franksdank/alloy:latest; then
+        skip "franksdank/alloy image not present on this box"
+    fi
+    local cfg="$BATS_TEST_TMPDIR/rendered.alloy"
+    sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
+    podman run --rm -v "$cfg:/c.alloy:ro" --entrypoint /usr/bin/alloy \
+        localhost/franksdank/alloy:latest validate /c.alloy
+}