docs(comfyui): the GTT aperture is not free RAM beside frank-dragoon

Measured 2026-10-21 with frank-dragoon live: MemAvailable 15.3 GiB, iGPU busy 1.75 GiB. The Qwen-Image weight sets (13.3-14.2 GB) do not fit that budget, and the page's 122 GB /system_stats figure is the aperture, not headroom. Adds the fit table and the stop-mask-render-verify procedure (the 3090 bench law, applied to the APU); ComfyUI.md gotcha links it.
docs(observability): post-blackbox state, copier law, label-promotion why
2026-10-21 13:35:50 -07:00 · 2026-10-21 13:33:03 -07:00 · 2026-10-21 13:33:03 -07:00 · 2026-10-21 13:33:03 -07:00 · 2026-10-21 13:33:03 -07:00 · 2026-10-21 13:32:58 -07:00
39 changed files with 2226 additions and 399 deletions
@@ -176,11 +176,12 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
 (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
 IP, the one bind a container can reach as `host.containers.internal`;
 iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
-Prometheus targets read down for structural reasons (HTML-scraped "probes",
-Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
-a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
-without `--metrics`. See
-`serverAppConfigs-docs/Infrastructure/Observability.md`.
+The former structural target downs (HTML-scraped "probes", Gitea metrics off)
+are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
+(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
+every Prometheus job reports up. `frank-zealot` was down for a different reason
+until 2026-10-18: llama.cpp will not serve `/metrics` at all without
+`--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.

 ## Testing

@@ -419,16 +420,17 @@ disagrees with it is the bug.
  digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
  Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
  `/home/dank/logs/frank_dragoon.log`, NOT the journal.
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
-  is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
-  `HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
-  the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
-  drafter always armed (the adaptive default disarms it and halves decode);
-  server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
-  The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
-  under-counts, so a harness window equal to the ceiling makes every long
-  request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
+- Shape: `HALOGEN_CTX=262144` (the checkpoint's native window) is the
+  **per-request ceiling** (a longer prompt is a 400, never a truncation);
+  `HALOGEN_KV_SLOTS=2` over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool —
+  pool positions buy concurrency, the per-request window never moves;
+  `HALOGEN_SPEC_ADAPT=0` keeps the MTP drafter always armed (the adaptive
+  default disarms it and halves decode); server sampler 1.0/0.95/20.
+- Harness contract: `models.yml` `contextWindow: 200000` — ~45k+ under the
+  ceiling. The clamp is `min(45056, window − est − 4096)` and the chars/4
+  estimate under-counts (trigger is 95 %), so a harness window at the ceiling
+  makes every long request a guaranteed 400.
+  Reasoning arrives in `reasoning_content`; the
  template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
  OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
  `/completion`, `/ping`, `/props`, `/tokenize` are 404.
@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
 Environment=DATABASE_PORT=5432
 Environment=USER_UID=1000
 Environment=USER_GID=1000
+# Prometheus scrapes /metrics over the published LAN port (job "gitea" in
+# shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
+# permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
+# the container; the only route in is this host's published 3000, and the
+# exposed surface is counters only — same posture as the LLM /metrics.
+Environment=GITEA__metrics__ENABLED=true
+Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
 PublishPort=0.0.0.0:3000:3000
 PublishPort=2222:22
 Volume=/home/dank/Containers/gitea/data:/data
@@ -0,0 +1,35 @@
+# Quadlet unit: blackbox exporter (rootless, boot-persistent).
+#
+# Install path on dankclaw:
+#   ~/.config/containers/systemd/blackbox-exporter.container
+#
+# WHY Network=host. The probes it runs are (a) the host's own LAN IP
+# (192.168.0.143:<port>) and (b) container names on dankclaw_default (comfyui).
+# A bridge container cannot reach the host's LAN IP at an unpublished port
+# (netavark DNATs published ports only — see Container Host Reachability in the
+# vault); a host-netns process reaches both. Prometheus (bridge) reaches THIS
+# container the same way it reaches the signal relay: the quadlet-published
+# host.containers.internal address (169.254.1.2), mapped via HostAddr= on the
+# prometheus unit.
+#
+# The 9115 metrics port publishes on loopback only — it is scraped by the
+# prometheus container through the gateway, and never opened on the LAN.
+
+[Unit]
+Description=Blackbox exporter (HTTP probe prober)
+After=network-online.target
+
+[Container]
+ContainerName=blackbox-exporter
+Exec=/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/config.yml
+Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+HealthCmd=/bin/sh -c "wget -q -O /dev/null http://127.0.0.1:9115/-/healthy || exit 1"
+HealthInterval=30s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,56 @@
+# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKCLAW.
+#
+# Install path on dankclaw: ~/.config/containers/systemd/cadvisor.container
+# Unit name: cadvisor.service (container name: cadvisor)
+#
+# WHY THIS EXISTS. The app stack is ~34 rootless Podman containers and until
+# now nothing reported per-container CPU/memory — Grafana could only see the
+# host (node_exporter) and the LLM engines. cadvisor reads the host cgroup
+# tree and exports container_cpu_* / container_memory_* per container.
+#
+# THE LOAD-BEARING FLAG IS --cgroupns=host. Podman starts every container in a
+# PRIVATE cgroup namespace, so an exporter container sees only its OWN cgroup
+# subtree: cadvisor exported a single root series until this flag was added,
+# then it walked the whole tree (996 per-container series measured 2026-10-20).
+# The quadlet unit name is carried in the cgroup path — app.slice/<name>.service/
+# libpod-payload-<hex> — so Prometheus relabels that path into a `container`
+# label (shared/prometheus/prometheus.yml, cadvisor-dankclaw job) and no
+# libpod/podman socket is needed to name containers.
+#
+# BINDING. cadvisor has no per-interface listen flag (unlike node_exporter's
+# --web.listen-address), so it binds 0.0.0.0 INSIDE its own network namespace.
+# The bridge PublishPort is what keeps the unauthenticated endpoint off the
+# host's other interfaces: it publishes exactly loopback (host-local probes and
+# the deployer's health gate) and this host's LAN address (Prometheus, a bridge
+# container, reaches a host-published port at its LAN address — the same netavark
+# rule node-exporter relies on, only here the port really is published).
+#
+# It is rootless on purpose: dank owns the app containers' cgroups, so a
+# rootless cadvisor reads every one of them; root would buy nothing here.
+
+[Unit]
+Description=cAdvisor container metrics (Dankclaw)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=cadvisor
+Image=gcr.io/cadvisor/cadvisor:v0.49.1
+PodmanArgs=--pid=host --cgroupns=host
+PublishPort=127.0.0.1:8080:8080
+PublishPort=192.168.0.143:8080:8080
+Volume=/sys:/sys:ro
+Volume=/:/rootfs:ro
+Exec=--housekeeping_interval=30s
+HealthCmd=wget --spider -q http://localhost:8080/healthz
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=40s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target
@@ -45,6 +45,11 @@ Wants=network-online.target
 ContainerName=prometheus
 Image=docker.io/prom/prometheus:latest
 Network=dankclaw_default
+# Name the podman gateway so scrape jobs can reach host-netns processes (the
+# blackbox exporter listens on the host's loopback; a bridge container reaches
+# it at this address, never at 127.0.0.1). Same mechanism the hermes Caddy
+# upstream learned the hard way (Container Host Reachability in the vault).
+HostAddr=host.containers.internal=169.254.1.2
 User=0:0
 Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
 PublishPort=127.0.0.1:9090:9090
@@ -0,0 +1,39 @@
+# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKSERVER.
+#
+# Install path on dankserver: ~/.config/containers/systemd/cadvisor.container
+# Unit name: cadvisor.service (container name: cadvisor)
+#
+# The Dankserver twin of dankclaw/observability/quadlet/cadvisor.container.
+# Same rootless shape, same --cgroupns=host requirement (see that file for the
+# full story), only the LAN address differs: this host publishes on
+# 192.168.0.235:8080 so Prometheus on Dankclaw can scrape it over the real
+# network (cross-host, so the host.containers.internal trick does not apply).
+# It reports the media/library stack (romm, plex, jellyfin, copyparty, calibre,
+# tautulli) by their quadlet unit names via the cgroup path.
+
+[Unit]
+Description=cAdvisor container metrics (Dankserver)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=cadvisor
+Image=gcr.io/cadvisor/cadvisor:v0.49.1
+PodmanArgs=--pid=host --cgroupns=host
+PublishPort=127.0.0.1:8080:8080
+PublishPort=192.168.0.235:8080:8080
+Volume=/sys:/sys:ro
+Volume=/:/rootfs:ro
+Exec=--housekeeping_interval=30s
+HealthCmd=wget --spider -q http://localhost:8080/healthz
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=40s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target
@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target

 [Container]
-Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
+Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329
 ContainerName=plex
 Network=dankserver_appnet:alias=plex
 Environment=PUID=0
@@ -34,12 +34,16 @@ SERVICE_ORDER = (
    "hermes",
    "hermes-dashboard",
    "hermes-webui",
-    "prometheus",
    "grafana",
    "loki",
    "alloy",
    "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
    "alertmanager",
+    # The prober must answer before prometheus re-runs the services job.
+    "blackbox-exporter",
+    "prometheus",
    "authentik-postgres",
    "authentik-server",
    "authentik-worker",
@@ -92,7 +96,10 @@ DANKCLAW_SERVICES = {
    "loki",
    "alloy",
    "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
    "alertmanager",
+    "blackbox-exporter",
    "authentik-postgres",
    "authentik-server",
    "authentik-worker",
@@ -124,6 +131,8 @@ DANKSERVER_SERVICES = {
    "readarr",
    "frank-archon",
    "node-exporter",
+    "cadvisor",
+    "gpu-exporter",
 }

 # FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
@@ -271,6 +280,8 @@ def services_for_path(path: str, target: str) -> set[str]:
            # Provisioned datasources and dashboards: git is the source of
            # truth, so a change here must restart the service that reads them.
            _add(services, "grafana")
+        elif group == "blackbox":
+            _add(services, "blackbox-exporter")
        elif group == "homepage":
            # Homepage is a Dankclaw-owned deployment. Shared homepage files
            # are intentionally not copied during a Dankserver-only run.
@@ -374,7 +385,7 @@ def services_for_path(path: str, target: str) -> set[str]:
            # consumer (prometheus before grafana, loki before alloy).
            if len(parts) >= 4 and parts[2] == "quadlet":
                name = parts[3].removesuffix(".container")
-                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
+                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
                    _add(services, name)
            return services
        if len(parts) >= 2 and parts[1] == "gitea":
@@ -417,8 +428,8 @@ def services_for_path(path: str, target: str) -> set[str]:
            # node-dankserver job had nothing to scrape.
            if len(parts) >= 4 and parts[2] == "quadlet":
                name = parts[3].removesuffix(".container")
-                if name == "node-exporter":
-                    _add(services, "node-exporter")
+                if name in {"node-exporter", "cadvisor"}:
+                    _add(services, name)
            return services
        # frank-archon's quadlet-era directory left the tree on 2026-09-26
        # (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
@@ -449,6 +460,12 @@ def services_for_path(path: str, target: str) -> set[str]:
            # change to it redeploying alertmanager is the only path that
            # reaches it on the host.
            _add(services, "alertmanager")
+        if name in {"gpu-exporter.py", "gpu-exporter.service"}:
+            # The host-process GPU exporter is installed directly by the
+            # gpu-exporter deploy on BOTH hosts, so its script and unit select
+            # that service (target_services then keeps it only on the host being
+            # planned). Editing the collector restarts the running exporter.
+            _add(services, "gpu-exporter")
        if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
            # llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
            # files here; each one selects frank-zealot only. The ik_llama.cpp
@@ -86,7 +86,7 @@ service_is_allowed() {
        dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
        dankclaw:comfyui|dankclaw:hermes|\
        dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:alertmanager|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|\
        dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
        dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -97,7 +97,7 @@ service_is_allowed() {
        dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
        dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
        dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
-        dankserver:frank-archon|dankserver:node-exporter|\
+        dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
        frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
        frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
        frankbook-pro:readarr)
@@ -275,14 +275,81 @@ install_signal_relay() {
    return 1
 }

+# Host-process GPU metrics exporter (amdgpu sysfs + nvidia-smi). Not a quadlet:
+# it runs python3 from the checkout on the host so it can read the DRM/cgroup
+# sysfs and shell out to nvidia-smi. Same two-address bind posture as
+# node-exporter/cadvisor (loopback + this host's LAN), so it is never a wildcard.
+deploy_gpu_exporter() {
+    local unit_dir="$HOME/.config/systemd/user" lan dropin i
+    case "$TARGET" in
+        dankclaw) lan=192.168.0.143 ;;
+        dankserver) lan=192.168.0.235 ;;
+        *) echo "gpu-exporter: unknown target '$TARGET'" >&2; return 2 ;;
+    esac
+    mkdir -p "$unit_dir"
+    cp scripts/gpu-exporter.service "$unit_dir/gpu-exporter.service"
+    dropin="$unit_dir/gpu-exporter.service.d"
+    mkdir -p "$dropin"
+    printf '[Service]\nEnvironment=GPU_EXPORTER_BINDS=127.0.0.1,%s\n' "$lan" \
+        > "$dropin/bind.conf"
+    systemctl --user daemon-reload
+    systemctl --user enable gpu-exporter.service
+    systemctl --user restart gpu-exporter.service
+    # The interpreter needs a moment to bind after restart (the same race that
+    # once failed a healthy signal-relay deploy).
+    for i in $(seq 1 15); do
+        curl -fsS -m 5 http://127.0.0.1:9401/metrics >/dev/null && return 0
+        sleep 1
+    done
+    echo "gpu-exporter is not answering on 127.0.0.1:9401/metrics" >&2
+    return 1
+}
+
 copy_loki_config() {
    run_privileged mkdir -p /opt/loki/config
    run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
 }

 copy_alloy_config() {
-    run_privileged mkdir -p /opt/alloy/config
-    run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
+    # The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
+    # carries the host's name, so Loki streams say WHICH machine's journal a
+    # line came from without a second config file per host.
+    # *_CONF_ROOT are test seams: the bats cases execute these copiers for
+    # real against a sandbox with a stubbed run_privileged.
+    local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
+    tmp="$(mktemp)"
+    sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
+    if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
+        echo "alloy config: host-label substitution failed" >&2
+        rm -f "$tmp"
+        return 1
+    fi
+    # Syntax gate: alloy's config language has no schema — a misplaced
+    # attribute (e.g. `on_error` outside the json stage) parses only at RUN
+    # time, and the unit then crash-loops with a healthy-looking deploy.
+    # Validate against the very image that will run it, before installing.
+    # Skipped only in the bats sandbox (no image there).
+    if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
+        if ! podman run --rm -v "$tmp:/c.alloy:ro" \
+                --entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
+                validate /c.alloy; then
+            echo "alloy config: validate FAILED against the pinned image" >&2
+            rm -f "$tmp"
+            return 1
+        fi
+    fi
+    run_privileged mkdir -p "$root/config"
+    run_privileged cp "$tmp" "$root/config/config.alloy"
+    rm -f "$tmp"
+}
+
+copy_blackbox_config() {
+    # Single-file mount target: the unit binds this exact path into
+    # /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
+    # BLACKBOX_CONF_ROOT is the same test seam.
+    local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
+    run_privileged mkdir -p "$root"
+    run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
 }

 copy_grafana_provisioning() {
@@ -291,14 +358,25 @@ copy_grafana_provisioning() {
    # datasource would be invisible to review, lost with the data volume, and
    # unknown to anyone reading the repository. The unit already mounts
    # /opt/grafana/provisioning read-only, so this only has to populate it.
-    run_privileged mkdir -p /opt/grafana/provisioning/datasources \
-        /opt/grafana/provisioning/dashboards
+    # GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
+    # copy runs unprivileged (the bats case executes this function for real).
+    local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
+    run_privileged mkdir -p "$root/provisioning/datasources" \
+        "$root/provisioning/dashboards"
    run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
-        /opt/grafana/provisioning/datasources/prometheus.yml
+        "$root/provisioning/datasources/prometheus.yml"
    run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
-        /opt/grafana/provisioning/dashboards/provider.yml
-    run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
-        /opt/grafana/provisioning/dashboards/llm-fleet.json
+        "$root/provisioning/dashboards/provider.yml"
+    # EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
+    # turns each file in the directory into a dashboard, so a copier that names
+    # files one by one silently drops new boards (measured 2026-10-21: the
+    # containers board existed in the repo since its commit and was NEVER
+    # provisioned — the panel-census browser check could not find it while the
+    # hand-placed older boards happened to be on disk). Same rule as the quadlet
+    # tier: the directory is the contract, the copy is generated state.
+    for dash in shared/grafana/provisioning/dashboards/*.json; do
+        run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
+    done
 }

 render_caddyfile() {
@@ -520,6 +598,11 @@ deploy_observability() {
            ;;
        node-exporter)
            ;;
+        cadvisor)
+            ;;
+        blackbox-exporter)
+            copy_blackbox_config
+            ;;
        *)
            echo "unknown observability service: $service" >&2
            return 2
@@ -533,6 +616,8 @@ deploy_observability() {
        grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
        loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
        node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
+        cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
+        blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
    esac
 }

@@ -1368,9 +1453,10 @@ deploy_anubis() {
 # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
 # qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
 # + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
-# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
-# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
-# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
+# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool at the
+# native ctx 262144 ceiling (raised 2026-10-21 from 200704 so the harness can
+# run a 200,000 window with its ~45k fit gap; the pool still carries two
+# 200k conversations at once), image input — on the Strix Halo iGPU. It stays a
 # ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
 # relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
 # file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
@@ -2000,7 +2086,8 @@ deploy_service() {
        plex) deploy_plex ;;
        tautulli) deploy_tautulli ;;
        alloy) deploy_alloy ;;
-        prometheus|grafana|loki|node-exporter|alertmanager) deploy_observability "$service" ;;
+        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
+        gpu-exporter) deploy_gpu_exporter ;;
        comfyui) deploy_comfyui ;;
        *) deploy_compose_service "$service" ;;
    esac
@@ -133,13 +133,15 @@ fi
 valid_service() {
    local service="$1"
    case "$TARGET:$service" in
+        # dankclaw:searxng-network is client-only: a network quadlet with no
+        # remote dispatch handler, so the remote gate does not (must not) list it.
        dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
        dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
        dankclaw:gitea-server|dankclaw:homepage|\
        dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
        dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
        dankclaw:alertmanager|\
-        dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
+        dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
        dankclaw:open-notebook|\
        dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
        dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
        dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
        dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
-        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
-        dankserver:frank-archon|\
+        dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
+        dankserver:prowlarr|dankserver:frank-archon|\
        frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
        frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
        frankbook-pro:readarr)
@@ -49,14 +49,21 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
 # stays at the image default, which is what was qualified) ------------------
 # Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
 HALOGEN_MODEL_ID=frank-dragoon
-# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
-# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
-# the page) — a window equal to the ceiling makes every long request a 400.
-HALOGEN_CTX=200704
-# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
-# at once. Pool positions buy concurrency; the per-request window never moves
-# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
-# RAM the pool takes is RAM the n-gram table's page cache loses.
+# PER-REQUEST ceiling, set to the checkpoint's NATIVE window (262,144 — a
+# longer prompt is a 400 naming the limit, never a truncation; stretching
+# past native would need HALOGEN_ROPE_YARN and is not on plan). The harness
+# `contextWindow` must sit ~45k under THIS number (the 400-zone law on the
+# page): with the window at 200,000 the gap is 62,144, which absorbs the
+# chars/4 under-count. A window at or near the ceiling makes every long
+# request a guaranteed 400.
+HALOGEN_CTX=262144
+# ONE pool behind both slots. The harness window is 200,000, so 401,408
+# still carries TWO maximum harness-length conversations at once
+# (2 x 200,000 + one 45,056 budget <= pool + cap headroom); two literal
+# ceiling-length requests would queue instead — buying those means raising
+# the pool, and RAM the pool takes is RAM the n-gram table's page cache
+# loses. Pool positions buy concurrency; the per-request window never moves
+# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it).
 HALOGEN_KV_POOL_POSITIONS=401408
 # Two concurrent conversations: the MTP drafter is not slot-locked on this
 # engine, so the second slot is capacity, not a speculation tax.
@@ -107,10 +114,13 @@ HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
 FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
 # Node paths and the XRT directory are variables so the bats seam can drive
 # these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
-# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
-# once at its own path, because those libraries find each other at their install
-# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
-# system dir, which dangle inside the container.
+# is mounted FILE BY FILE and THREE TIMES EACH — where the image looks for XRT
+# (/opt/xilinx/xrt/lib), at its own path (those libraries find each other at
+# their install path), and at /opt/xilinx/lib/<multiarch>/ (the NPU child
+# searches that third prefix; without it the lane dies naming it and the
+# supervisor takes the container down — 2026-10-21 boot loop). The basename of
+# this directory IS the multiarch name. The vendor forbids mounting an
+# /opt/xilinx/xrt of symlinks into the system dir, which dangle in the container.
 FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
 FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
 # sysfs root the fabric-clock preflight reads (same knob the vendor's
@@ -1,5 +1,5 @@
 [Unit]
-Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
+Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 262144, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
 After=network-online.target
 Wants=network-online.target

@@ -0,0 +1,212 @@
+#!/usr/bin/env python3
+"""gpu-exporter: unified per-GPU metrics for a mixed AMD/NVIDIA host.
+
+Emits the `gpu_*` Prometheus families the host-system dashboard graphs:
+
+    gpu_utilization_percent  gpu_memory_used_bytes  gpu_memory_total_bytes
+    gpu_temperature_celsius  gpu_power_watts
+
+each labelled {gpu, vendor, model}. The scrape job adds `host`.
+
+Two sources, both host-native — no container, no GPU passthrough:
+
+  * AMD (R9700s, the 8060S iGPU, the Raphael display GPU): the amdgpu sysfs tree.
+    Utilization comes ONLY from gpu_busy_percent — amd-smi reports gfx_busy_inst
+    as N/A on gfx1201 (verified 2026-10-20). VRAM is already bytes, temperature
+    is milli-degrees, power is micro-watts. Crucially power1_average is EMPTY on
+    the R9700s, so those cards legitimately have no gpu_power_watts line; a
+    missing file becomes an ABSENT metric, never a fabricated 0 that a dashboard
+    would plot as a real reading.
+
+  * NVIDIA (the RTX 3090 on Dankclaw): one `nvidia-smi --query-gpu` CSV. On
+    Dankserver there is an nvidia-smi binary but no NVIDIA GPU, so the command
+    fails; that is tolerated and simply yields no NVIDIA series.
+
+Default mode prints one scrape to stdout and exits (the test harness, and any
+`service ... --oneshot` one-off). `--serve` binds the HTTP endpoint for
+Prometheus; it binds EXPLICITLY (loopback + the host LAN address), never a
+wildcard, matching node-exporter's posture on these boxes.
+"""
+import argparse
+import glob
+import os
+import subprocess
+import sys
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+FAMILIES = [
+    ("gpu_utilization_percent", "gauge", "GPU busy time as a percentage of wall time."),
+    ("gpu_memory_used_bytes", "gauge", "Device memory in use, in bytes."),
+    ("gpu_memory_total_bytes", "gauge", "Device memory capacity, in bytes."),
+    ("gpu_temperature_celsius", "gauge", "Package temperature in degrees Celsius."),
+    ("gpu_power_watts", "gauge", "Board power draw in watts."),
+]
+
+DEFAULT_NVIDIA_CMD = (
+    "nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,"
+    "memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits"
+)
+
+
+def _read(path):
+    try:
+        with open(path) as fh:
+            return fh.read().strip()
+    except OSError:
+        return ""
+
+
+def _pci_slot(device_dir):
+    uevent = _read(os.path.join(device_dir, "uevent"))
+    for line in uevent.splitlines():
+        if line.startswith("PCI_SLOT_NAME="):
+            return line.split("=", 1)[1]
+    return ""
+
+
+def _pci_id(device_dir):
+    uevent = _read(os.path.join(device_dir, "uevent"))
+    for line in uevent.splitlines():
+        if line.startswith("PCI_ID="):
+            return "amd:" + line.split("=", 1)[1].lower()
+    return "amd"
+
+
+def collect_amd(sys_dir):
+    for card in sorted(glob.glob(os.path.join(sys_dir, "card[0-9]*"))):
+        device = os.path.join(card, "device")
+        busy = _read(os.path.join(device, "gpu_busy_percent"))
+        if busy == "":
+            continue  # not an amdgpu compute node (e.g. a render node w/o metrics)
+        labels = {"gpu": _pci_slot(device) or os.path.basename(card),
+                  "vendor": "amd", "model": _pci_id(device)}
+        yield "gpu_utilization_percent", labels, busy
+        for name, leaf in (("gpu_memory_used_bytes", "mem_info_vram_used"),
+                           ("gpu_memory_total_bytes", "mem_info_vram_total")):
+            v = _read(os.path.join(device, leaf))
+            if v != "":
+                yield name, labels, v
+        temp = ""
+        for hwmon in sorted(glob.glob(os.path.join(device, "hwmon", "hwmon*", "temp1_input"))):
+            temp = _read(hwmon)
+            if temp != "":
+                break
+        if temp != "":
+            yield "gpu_temperature_celsius", labels, str(int(temp) / 1000.0)
+        power = _read(os.path.join(device, "power1_average"))
+        if power != "":  # empty on the R9700s -> metric absent, not zero
+            yield "gpu_power_watts", labels, str(int(power) / 1e6)
+
+
+def collect_nvidia(cmd):
+    if not cmd:
+        return
+    try:
+        out = subprocess.run(cmd, shell=True, capture_output=True, text=True,
+                             timeout=10, check=True).stdout
+    except (subprocess.SubprocessError, OSError):
+        return  # no NVIDIA GPU / no driver: tolerate, emit nothing
+    for line in out.splitlines():
+        parts = [p.strip() for p in line.split(",")]
+        if len(parts) < 7:
+            continue
+        idx, name, util, mused, mtotal, temp, power = parts[:7]
+        labels = {"gpu": idx, "vendor": "nvidia", "model": name}
+        yield "gpu_utilization_percent", labels, util
+        yield "gpu_memory_used_bytes", labels, str(int(float(mused)) * 1048576)
+        yield "gpu_memory_total_bytes", labels, str(int(float(mtotal)) * 1048576)
+        if temp not in ("", "[N/A]"):
+            yield "gpu_temperature_celsius", labels, temp
+        if power not in ("", "[N/A]"):
+            yield "gpu_power_watts", labels, power
+
+
+def collect(sys_dir, nvidia_cmd):
+    samples = {}
+    for source in (collect_amd(sys_dir), collect_nvidia(nvidia_cmd)):
+        for name, labels, value in source:
+            try:
+                value = str(float(value))  # normalise; drop trailing junk
+            except ValueError:
+                continue
+            samples.setdefault(name, []).append((labels, value))
+    return samples
+
+
+def _fmt(value):
+    f = float(value)
+    if f == int(f) and abs(f) < 1e15:
+        return str(int(f))
+    return repr(f)
+
+
+def render(samples):
+    out = []
+    for name, kind, help_text in FAMILIES:
+        out.append("# HELP %s %s" % (name, help_text))
+        out.append("# TYPE %s %s" % (name, kind))
+        for labels, value in sorted(samples.get(name, []), key=lambda x: sorted(x[0].items())):
+            lset = ",".join('%s="%s"' % (k, v) for k, v in sorted(labels.items()))
+            out.append("%s{%s} %s" % (name, lset, _fmt(value)))
+    return "\n".join(out) + "\n"
+
+
+class Handler(BaseHTTPRequestHandler):
+    sys_dir = "/sys/class/drm"
+    nvidia_cmd = DEFAULT_NVIDIA_CMD
+
+    def do_GET(self):  # noqa: N802 (http.server API)
+        if self.path.split("?")[0] not in ("/metrics", "/"):
+            self.send_error(404)
+            return
+        body = render(collect(self.sys_dir, self.nvidia_cmd)).encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "text/plain; version=0.0.4")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args):
+        pass
+
+
+def main(argv):
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--amdgpu-sysfs", default="/sys/class/drm")
+    ap.add_argument("--nvidia-cmd", default=DEFAULT_NVIDIA_CMD)
+    ap.add_argument("--serve", action="store_true", help="serve /metrics instead of one-shot")
+    ap.add_argument("--port", type=int, default=9401)
+    ap.add_argument("--bind", action="append", default=None,
+                    help="explicit bind address (repeatable); env GPU_EXPORTER_BINDS "
+                         "comma-list if unset. NEVER a wildcard.")
+    args = ap.parse_args(argv)
+
+    if not args.serve:
+        sys.stdout.write(render(collect(args.amdgpu_sysfs, args.nvidia_cmd)))
+        return 0
+
+    binds = args.bind
+    if not binds:
+        env = os.environ.get("GPU_EXPORTER_BINDS", "")
+        binds = [b.strip() for b in env.split(",") if b.strip()] or ["127.0.0.1"]
+    Handler.sys_dir = args.amdgpu_sysfs
+    Handler.nvidia_cmd = args.nvidia_cmd
+    servers = []
+    for b in binds:
+        if b in ("0.0.0.0", "::", "*"):
+            sys.stderr.write("refusing wildcard bind %s\n" % b)
+            return 2
+        servers.append(ThreadingHTTPServer((b, args.port), Handler))
+    sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
+    # EVERY socket must be served: a listen()ed-but-never-accept()ed bind looks
+    # healthy in `ss` yet times out every scrape once its SYN backlog fills —
+    # and the loopback deploy gate cannot see it. One thread per extra bind.
+    for s in servers[1:]:
+        threading.Thread(target=s.serve_forever, daemon=True).start()
+    servers[0].serve_forever()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))
@@ -0,0 +1,35 @@
+# Systemd USER unit: gpu-exporter (both Dankclaw and Dankserver).
+#
+# Install path: ~/.config/systemd/user/gpu-exporter.service
+# Installed and enabled by deploy_gpu_exporter in scripts/deploy-remote.sh.
+#
+# Per-GPU metrics (utilization / VRAM / temperature / power) for the mixed
+# AMD + NVIDIA fleet, scraped by Prometheus. Like signal-relay it is a plain
+# user unit, not a quadlet: it runs python3 straight from the checkout (a pull
+# updates it with no copy step), a hand-written user unit can be enabled where
+# the podman v1 generator's units cannot, and with Linger=yes for dank it starts
+# at boot. It reads the host's own cgroup/drm sysfs and shells to nvidia-smi, so
+# it must run on the host — a container would need GPU/DRM passthrough for
+# nothing gained.
+#
+# BINDING. The base unit serves loopback only. deploy_gpu_exporter writes a
+# drop-in adding THIS host's LAN address (the same two-address posture
+# node-exporter and cadvisor use), so the endpoint is never a wildcard. It is
+# unauthenticated metrics on the LAN, identical to node-exporter :9100 and
+# cadvisor :8080 — Prometheus reaches same-host via host.containers.internal and
+# the Dankserver instance over the LAN from Dankclaw.
+
+[Unit]
+Description=GPU metrics exporter (amdgpu sysfs + nvidia-smi)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+# Loopback by default; the deploy drop-in appends the host LAN address.
+Environment=GPU_EXPORTER_BINDS=127.0.0.1
+ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target
@@ -136,10 +136,17 @@ fi
 #      The engine re-checks size+sha256 against that catalog at boot; it cannot
 #      fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
 #      stays unset — no outbound connection at boot),
-#   2. the distro XRT's three libraries, each mounted twice (where the image
-#      looks for XRT, and at its own path — the libraries find each other there;
-#      an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
-#      container, which the vendor calls out explicitly),
+#   2. the distro XRT's three libraries, each mounted THREE times: where the
+#      image looks for XRT (/opt/xilinx/xrt/lib), at their own path (the
+#      libraries find each other there), and at /opt/xilinx/lib/<multiarch>
+#      — the NPU child searches that third prefix too, and since the host's
+#      libxrt2 was reconfigured 2026-10-20 it is the ONLY prefix the child
+#      names: omitting it crashes the lane with `No such library
+#      '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the engine
+#      supervisor takes the dead lane down on the whole container (measured
+#      2026-10-21: 88-restart boot loop). Real files only — an /opt/xilinx/xrt
+#      of symlinks into the system dir dangles inside the container, which the
+#      vendor calls out explicitly),
 #   3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
 #      hang the machine while FCLK changes speed, and the engine refuses to
 #      start when it moves — it cannot read host sysfs from inside the
@@ -190,7 +197,9 @@ if [ -n "$NPU_MODELS" ]; then
            exit 1
        fi
        REAL="$(readlink -f "$XRT_DIR/$L")"
-        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
+        MULTIARCH="${XRT_DIR##*/}"
+        NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro"
+                   -v "$REAL:/opt/xilinx/lib/$MULTIARCH/$L:ro")
    done
    HELD=0
    for D in "$SYSFS"/class/drm/card*/device; do
@@ -293,6 +293,14 @@ for path in "${CHANGED[@]}"; do
            add_test tests/caddyfile.bats
            add_test tests/deployment_scoping.bats
            ;;
+        # The GPU exporter is a host process; its seam pins the collector, the
+        # user unit, the deployer function, the planner route, the Prometheus
+        # scrape jobs and the host-system GPU panels — a cross-file contract, so
+        # the whole seam rides on any of its own files.
+        scripts/gpu-exporter.py|scripts/gpu-exporter.service)
+            add_test tests/gpu_exporter.bats
+            add_test tests/deployment_scoping.bats
+            ;;
        dankclaw/udev/*)
            # Host-level udev rules are applied by manual sudo install + udevadm
            # trigger, not dispatched as a service; nothing here is bats-testable.
@@ -52,28 +52,31 @@ Where the contract lives:
 The arithmetic (the coupled contract every endpoint's maxTokens comment
 encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
 endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
-follows is a **worked example** recomputed from the live files on 2026-10-18 —
-a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
+follows is a **worked example** recomputed from the live files on 2026-10-21 —
+a snapshot, not fleet state — with windows 200,000 / 200,000 / 200,704 and
 maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):

 1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
-   95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
+   95 % that is 190,000 (archon and dragoon), 190,669 (zealot). The
   bundle clamps the percentage to 1–99 and converts it to a token threshold.
 2. **Output budget** = capped at the target model's `maxTokens` — every role
   here compacts to frank-zealot, so the budget is its 16,384.
-3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
-   window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
-   200,704. The registry's own comments design around an 85 % trigger (archon:
-   170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
-   does hold. While `config.yml` and those comments disagree, treat 85 % as the
-   design point and re-run this arithmetic after ANY window change — a 400 from
-   the compaction target mid-conversation is the symptom.
+3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the fitting window. For a
+   dragoon session the fitting window is the ENGINE ceiling, not a slot: the
+   190,000 trigger is 62,144 under the 262,144 ceiling, so even ×1.03 est
+   error plus the 45,056 request budget lands inside — that headroom is the
+   reason the ceiling moved to native on 2026-10-21. For a session compacting
+   INTO frank-zealot the window is its 200,704-token slot, and **at 95 % this
+   does not hold for archon/zealot sources**: 190,000 + 16,384 = 206,384 >
+   200,704. Treat 85 % as the design point until config.yml and the registry
+   comments agree, and re-run this arithmetic after ANY window change — a 400
+   from the compaction target mid-conversation is the symptom.
   (Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
   177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
   32,768 ≤ 200,704 zealot.)

 Procedure (recompute from the LIVE files every time — windows rotate with
-every swap; the block above is the 2026-10-18 state):
+every swap; the block above is the 2026-10-21 state):

 1. Read the `config.yml` compaction block + `models.yml` entries for source
   and target. Every endpoint's `contextWindow` comment carries its own
@@ -94,9 +97,10 @@ Known reconciliation trap: comments in models.yml have described compaction
 as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
 dankclaw — the comment/config disagreed. The comments now name the real
 target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
-truth; fix comments in the same change.** A fresh instance of the same trap is
-open 2026-10-18: every `maxTokens` comment still says "compaction fires at
-85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
+truth; fix comments in the same change.** The 85 %-comments vs 95 %-config
+disagreement is not academic: it is why long dragoon sessions kept drawing
+400s at the old 200,704 ceiling — the trigger fires late AND the est
+under-counts, and a window parked near the ceiling has no room for either.
 `maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
 input. Boot-shape OOM laddering is a different problem
 ([[Llama Cpp Serving Sizing]]).
@@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** —
 docs, unit docstrings, and the repo's rule file (the most expensive place to
 be stale).

+## Three rules that outlive the incident that taught them
+
+* **A target can read down beside a healthy service.** Read `lastError` from
+  `/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the
+  protocol layer without `--metrics` (501) while `/health` answers 200 for weeks.
+* **Alert state lives where the tool says it does.** Prometheus keeps rule state in
+  process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is
+  not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a
+  host volume: silences + notification log, [[Alertmanager]]).
+* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production
+  unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape,
+  [[frank-zealot]]) beside an answering port means the metric measured a scratch boot;
+  the process shape can change between two censuses ten minutes apart.
+
 ## Order and verification

 1. Enumerate the surfaces (status board, metrics targets, alert rules, test
@@ -110,9 +110,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file

 ## Recommended additions

-Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
-reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
-2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
+Quantized preferred. The iGPU's GTT aperture is not free RAM beside
+frank-dragoon — size every pick against *What actually fits beside
+frank-dragoon* below (the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
 all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
 `huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
 `huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
@@ -133,8 +133,34 @@ sha256 checks) or a plain `hf download` with the token in the environment —
 never on the command line.

 Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
-the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
-them at GTT capacity, and they double the disk bill).
+the 2.1 files already on disk), and any BF16 of #2/#4 — they double the disk
+bill, and see *What actually fits beside frank-dragoon* for why raw GTT size
+is not the real budget.
+
+## What actually fits beside frank-dragoon
+
+The 122 GB `/system_stats` figure is the GTT **aperture**, not free memory: on
+an APU the aperture and `MemAvailable` are the same physical RAM, and
+frank-dragoon pins tens of GiB of it. Measured 2026-10-21 with frank-dragoon
+live: `MemAvailable` 15.3 GiB, iGPU busy 1.75 GiB — ComfyUI itself idles at
+~10 MB, so the budget for a generation run is roughly what `free` says.
+
+Weights must land inside that budget with slack for compute buffers and
+activations (the mmproj-class lesson: price a model at ~1.5× its file set):
+
+| Pipeline (weights) | Fits beside frank-dragoon? |
+|---|---|
+| Qwen-Image 2.1 int8_convrot + w4a8 + bf16 VAE (~14.2 GB) — the two proven workflows | **No** — thrash-or-fail; do not queue it while the endpoint serves |
+| GGUF Q8_0 + w4a8 + small VAE (~13.3 GB) | No, same reason |
+| Any BF16 combination | No |
+| SDXL-Turbo / SD 1.5-class (~2–7 GB) | Yes, with real headroom |
+
+**The law: one process owns the memory — same rule as the 3090 bench law.**
+For a Qwen-Image render: stop and mask frank-dragoon
+(`sudo systemctl stop frank-dragoon` + `mask` — the mask stops
+`Restart=always` from firing mid-render), generate, unmask, verify
+`curl 192.168.0.143:7770/health`. The proven workflows above then run with
+the whole machine. Do not treat `/system_stats`'s 122 GB as an invitation.

 ## See Also

@@ -129,6 +129,12 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
 - Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
 - The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
 - The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
+- **The 122 GB `/system_stats` VRAM figure is the GTT aperture, not free
+  memory.** On the APU it is system RAM, and frank-dragoon pins most of it
+  (`MemAvailable` 15.3 GiB with the endpoint live, measured 2026-10-21). A
+  Qwen-Image-class render beside the running endpoint is thrash-or-fail; the
+  stop-mask-render-verify procedure and the fit table are in
+  [[ComfyUI Qwen-Image Workflows]] → What actually fits beside frank-dragoon.

 ## See Also

@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
 tags:
  - dankserver
  - plex
-timestamp: 2026-09-15
+timestamp: 2026-10-21
 ---

 # Plex
@@ -24,7 +24,7 @@ public traffic can be fronted by Caddy like every other vhost.
 | Item | Value |
 |------|-------|
 | Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
-| Image | `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (pinned; verified == the ghcr `latest` manifest digest at pin time) |
+| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
 | Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
 | Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
 | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
@@ -3,7 +3,7 @@ type: reference
 title: Observability Provisioning
 description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
 tags: [observability, prometheus, grafana, provisioning, deployment]
-timestamp: 2026-10-05T00:00:00-07:00
+timestamp: 2026-10-21T00:00:00-07:00
 ---

 # Observability Provisioning
@@ -20,6 +20,8 @@ committed.
 | `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
 | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
 | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
+| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`) + the loopback bind |

 Targets and datasources reference **container names** (`grafana:3000`
 `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
 reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
 the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
 (`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
-HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
-[[Stale Signal Audit]]).
+HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
+deployed 2026-10-21; see [[Stale Signal Audit]]).

 ## The deploy path

@@ -43,8 +45,9 @@ from `deploy_observability`:
 - `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
 - `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
 - `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
+- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
+- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)

 `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
 `alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
@@ -82,11 +85,14 @@ reports 4 alerting + 2 recording. Rules reload with the prometheus service.
 the backend (`http://prometheus:9090`), not loopback.

 **Add a dashboard** — drop the JSON into
-`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
-unified storage, not (only) the SQLite table — grep the API, not the DB, when
-verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
-2026-10-18: the storage-backend claim needs an admin password to confirm
-against `grafana.db` and the API side by side)*
+`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
+the next `--service grafana` deploy provisions it. This is a repaired law, not
+a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
+`containers` board — merged, tested, documented as "working" — had **never
+been provisioned**; the other boards existed on disk only because someone had
+hand-copied them once. A repo JSON is not a live dashboard until the copier
+carries it, and `deploy.sh` cannot see the gap because it only reports the
+copy it performed.

 ## Verify through the APIs, not the filesystem

@@ -95,90 +101,32 @@ against `grafana.db` and the API side by side)*
 curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
 curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
-# grafana: datasource + dashboard provisioned
-curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
+# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
+# account is the Authentik RP login). The proven surface is a real browser
+# session: warm the Anubis challenge on the vhost root, then hit
+# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
+# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
 ```

 The two Prometheus queries were run on 2026-10-18 and are what every number on
 this page comes from: 4 alerting + 2 recording rules, and `up` for
 `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
-`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
-listed in [[Prometheus]]. The Grafana query needs the local admin password and
-was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
-proof of installation only — see the next paragraph.
+`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
+that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
+2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
+filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
+only — see *Add a dashboard* for why a repo JSON is not a live board.

 Health answers from a stale process and provisioned resources can hide in
 unified storage — a file existing on disk is not proof the running server
 loaded it. See [[Stale Signal Audit]] for the never-green-target decision
 table (refused = outage, 404/HTML = config defect).

-## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
-
-Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
-loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
-unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
-via `host.containers.internal`) → signal-cli, addressed by number
-`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
-2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
-`curl -s http://192.168.0.143:18093/health` → `ok`, and
-`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
-lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
-`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
-structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
-the blackbox exporter. See [[Prometheus]] → Targets.
-
-**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
-`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
-the oldest sample, 2026-10-11, is already 0 — while the service itself answered
-`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
-is started with the flag, so the scrape was refused at the protocol layer (the
-flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
-stayed firing:
-the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
-on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
-metrics opt-in landed that day; the target
-reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
-How many payload generations it spanned is not readable from any host
-artifact. *(unverified 2026-10-18)*
-
-Two laws fall out of it. A target can read down beside a healthy service, so
-read `lastError` from `/api/v1/targets` before believing an outage. And rule
-state lives in Prometheus' process, not on disk — a restart re-stamps
-`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
-evidence of a fresh fault. (Alertmanager by contrast **does** persist:
-`--storage.path=/alertmanager` on a host volume keeps silences and the
-notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
-`/opt/alertmanager/data`.)
-
-Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
-metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
-([[frank-dragoon]]), which is why
-`frank-dragoon` has always been green. The active-alert count is a
-moment-in-time value and should never be documented as a constant:
-`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
-`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
-minutes later, and was empty again by 11:56Z; the relay logged the matching
-firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
-
-**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
-production `frank-zealot` root unit was stopped **on purpose** for a benchmark
-window on the 3090 — the repo Benchmarking Policy says stop the production
-service before benching — and `:7771` answered from a manually booted candidate
-arm (`systemctl is-active frank-zealot` → `failed` after
-`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
-`TimeoutStopSec`, the known shutdown shape for these engines). So
-`up{job="frank-zealot"} == 1` that day measured the trial, not the production
-shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
-and it is the repo contract.
-The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
-reinstalls and starts it, restoring boot persistence and self-healing. Never
-quote a candidate arm's flags or VRAM as fleet state — the process and its shape
-changed between two censuses ten minutes apart.
-
 ## Related

 - [[Observability]] — stack topology and the alert delivery chain
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
+  down-beside-healthy / alert-state-persistence / candidate-arm rules
 - [[Quadlet Lifecycle]] — the units that host these services
 - [[Deploy State Triage]] — how the planner selects these services
@@ -2,7 +2,7 @@
 type: reference
 title: "Observability on Dankclaw"
 description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
-tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
+tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
 timestamp: 2026-10-20
 ---

@@ -10,10 +10,10 @@ timestamp: 2026-10-20

 ## Summary

-Six containers and one host-side process: Prometheus (metrics), Grafana
+Seven containers and two host-side processes: Prometheus (metrics), Grafana
 (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
-metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
-carrying every firing group to Signal. All of it is Quadlet-managed,
+metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
+plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
 boot-persistent and loopback-only, deployed by
 `scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
 end to end; the log half is configured, healthy, and carries no data.
@@ -34,6 +34,8 @@ that proves it today.
 | Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
 | Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
 | node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
+| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
+| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
 | Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |

 Every unit lives in `dankclaw/observability/quadlet/`, carries
@@ -44,9 +46,11 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
 the same day, and `homepage-status.timer` — the root unit that probes all of
 these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
 the probe registry `shared/homepage/services.json` to `/srv/homepage`).
-Dankserver runs one member of this family, node-exporter, as its own unit
-(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
-`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
+Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
+(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
+`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
+cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
+`gpu-exporter` (a user unit installed the same way as `signal-relay`).

 ## Exposure

@@ -63,10 +67,18 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
 in the host network namespace, and it binds exactly two addresses rather than
 the wildcard — see [[Node-Exporter]].

-The relay's bind is the one address a bridge container can reach on the host:
-`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
-and the podman bridge subnet `10.89.0.0/16` before the DROP
-(`scripts/deploy-remote.sh:255-262`).
+**Same-host reachability from a bridge container is measured, not inferred**
+(2026-10-21, from inside the Prometheus container): `host.containers.internal`
+(the podman gateway) reaches host services whether they bind host-net or are
+published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
+— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
+every same-host Prometheus target uses the gateway and only cross-host targets
+(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
+relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
+iptables INPUT admitting loopback, the host itself and the podman bridge subnet
+`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
+point a scrape target at a host LAN IP from its own host on the strength of
+one working example; dial the gateway and probe.

 ## Deployment

@@ -83,13 +95,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
 [[Observability Provisioning]] for what is provisioned as code and how to
 verify it through the APIs.

-## Current state (2026-10-20)
+## Current state (2026-10-21)

 * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
-  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
-  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
-  and both node exporters. The four downs are structural (the `comfyui` and
-  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
+  every job up — the three LLM endpoints (frank-archon on `vllm:*`,
+  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
+  both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
+  the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
+  downs are retired — see [[Prometheus]] → Targets.
  Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
  (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
  alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
@@ -97,30 +110,59 @@ verify it through the APIs.
 * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
  relay, and `journalctl --user -u signal-relay` records successful
  `sent to +12062476336:` lines for every rule in the set.
-* **Dashboards: working.** Provisioned datasources — Prometheus (uid
-  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
-  installed from `shared/grafana/provisioning/`. The Loki datasource is how
-  you read logs: Loki has no web UI of its own, so Grafana Explore is the
-  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
-* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
-  a rootless Alloy cannot read the `0640` journal files, and the container-log
-  globs point at a root-storage layout that does not exist here. The fix is built
-  and tested in-repo (2026-10-20): Alloy is now a **root system service on
-  `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
-  Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
-  lines and pushed 26,966 with zero drops. The **live** instance is still the old
-  rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
-  [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
-* **Not configured:** no blackbox exporter (the structural downs have no real
-  uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
-  not yet deployed (see *Logs* above).
+* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
+  datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
+  `llm-fleet`, `host-system`, `containers`, `observability` and
+  `logs-overview` boards. Until 2026-10-21 the copier carried only
+  `llm-fleet.json` by name and the `containers` board had never been
+  provisioned; the deployer now globs the directory. The Loki datasource is
+  how you read logs: Loki has no web UI of its own, so Grafana Explore is the
+  human surface (the guarded `loki.franksdank.cloud` vhost serves only the
+  API). Verify method and the copier law: [[Observability Provisioning]].
+* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
+  after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
+  per-container CPU/memory here, and on this rootless-Podman host it works on one
+  condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
+  namespace, so without that flag cadvisor sees only its own cgroup root and
+  exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
+  names nothing itself — the libpod socket resolved only 2 of 34 rootless
+  containers — but the quadlet unit name is carried in the cgroup path
+  (`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
+  `container` label. Network and filesystem are NOT exported for these containers
+  (no netns/classid attribution under the host-PID layout), so the dashboard
+  graphs only CPU, memory and OOM and does not fake the rest.
+* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
+  `deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
+  host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
+  own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
+  without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
+  `mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
+  emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
+  °C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
+  job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
+  `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
+  the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
+  node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
+* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
+  pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
+  the container-log globs pointed at a root-storage layout that does not exist
+  here) — fixed 2026-10-20 by making Alloy a **root system service on
+  `Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
+  but the stream carried only `job`/`service_name`, so every per-container
+  view was structurally empty: Loki metrics queries group only by INDEXED
+  stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
+  label (plus a `host` label) at ingestion — `loki.process` in
+  `shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
+  purpose-test law in [[Observability Provisioning]].

 ## Citations

- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
+- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
 - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
 - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
+- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
 - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
 - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
+- `shared/blackbox/blackbox.yml` (prober modules + loopback bind)
 - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
 - Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
@@ -42,29 +42,27 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —

 The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.

-**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
+**No job reads down structurally any more** (the three permanent-reds were
+retired 2026-10-21; census re-run after each deploy):

-| Job | Target | `health` | `lastError`, verbatim |
-|-----|--------|----------|------------------------|
-| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
-| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
+| Job | Target | how |
+|---|---|---|
+| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
+| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://comfyui:8188` | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service |

-Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
+Scraping a web root is not a health check: an HTML page fails the scrape
+protocol (`unsupported Content-Type`), which is what kept those targets red
+beside healthy services. The classification rule and the honesty law live in
+[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
+netns, config `shared/blackbox/blackbox.yml`, bound 127.0.0.1:9115; Prometheus
+reaches it at the `host.containers.internal` gateway name). Plain targets stay
+`host:port`, never URLs — the probe job's URL targets are the documented
+exception, relabeled onto the prober.

 **`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.

 Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.

-The remedy for a probe-shaped question is the **blackbox exporter**; the
-classification rule and the honesty law for it live in [[Stale Signal Audit]].
-Until the exporter is deployed, these three stay red for a structural reason:
-deleting them would
-hide a real service, and fixing them by publishing databases or enabling metrics
-would trade a wrong signal for a real exposure. Targets are `host:port`, never
-URLs.
-
 ## Volumes

 | Container path | Host path |
@@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in

 `--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
 reserves the registry price, not the actual pool. Raising the pool consumes
-real free VRAM without growing the arena; the registry over-prices the pool
-(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
-**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
-block-alignment loss when converting a token target to bytes). The ask lands
-block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
-`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
+real free VRAM without growing the arena; the registry OVER-PRICES the pool
+(era G measured ~14 % under the planned B/token — the gap is the lesson, not
+the number; **never reuse a B/token ratio across geometries**), and there is a
+~4 MB block-alignment loss to subtract when converting a token target to
+bytes. The ask lands block-aligned: verify against the engine's own
+`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot.

 **Driver-reported free VRAM after hours of traffic is NOT allocatable
 arena.** It is the estimator gap: the registry over-prices KV and charges
@@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
 **KV accounting — pin the pool BYTES, floor-sized:** set
 `--kv-cache-memory-bytes` PER RANK to the contract floor
 (`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
-come out of the arena/headroom and buy nothing. Measured B/token on this
-hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
-dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
-7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
-planner's KV reserve is for ONE max-model-len request.
+come out of the arena/headroom and buy nothing. The B/token rate is a property
+of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**;
+the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape.
+Read your live rate from the boot ledger every time; the decomposition (a
+dtype-independent mamba/side stream rides on top of the attention KV) is why
+the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve
+is for ONE max-model-len request.

 **Parallelism ceilings you cannot flag past:**

@@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness
 (`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
 + engine flags in the log.

-**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
-
-| shape | util | graphs | arena | result |
-|---|---|---|---|---|
-| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
-| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
-| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
-| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
-
-bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
-at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
-~207k per context max.

 ## 7. Arena audit procedure (when asked for "more experts")

@@ -10,14 +10,15 @@ tags:
  - gfx1151
  - halogen
  - endpoint
-timestamp: 2026-10-19
+timestamp: 2026-10-21
 ---

 # frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)

 Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
 under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
-(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
+(`2b12716`), ceiling raised to the native 262,144 on 2026-10-21 so the harness window is 200,000.
+This page replaces every per-era dragoon runbook; the Eras table is the rollback record.

 ## Current shape

@@ -37,10 +38,12 @@ Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b
 unauthenticated token port 8730 stays on container loopback. Engine stdout →
 `/home/dank/logs/frank_dragoon.log`, **not** the journal.

- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
-  `HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
-  enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
-  exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
+- Shape: `HALOGEN_CTX=262144` hard per-request ceiling = the checkpoint's native window (raised
+  2026-10-21 from 200704; `HALOGEN_ROPE_YARN` beyond native is not on plan) · `HALOGEN_KV_SLOTS=2` ·
+  `HALOGEN_KV_POOL_POSITIONS=401408` inside the band `tests/frank_dragoon_unit.bats` enforces
+  (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
+  exists here, the pool is the only KV lever · the pool carries TWO maximum harness-length (200k)
+  conversations at once; two literal ceiling-length requests would queue — that is a pool raise.
 - Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
  omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
  pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
@@ -53,10 +56,11 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
  (Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
  ~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
  Caddy/observability).
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
-  (= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
-  `reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
-  rides [[frank-zealot]].
+- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 200000`
+  — sits 62,144 under the ceiling, clearing the ~45k fit-law gap (400-zone law below);
+  `tests/frank_dragoon_unit.bats` fails if the page's declared window and `HALOGEN_CTX` drift back
+  together. `maxTokens: 45056`, `input: [text, image]`, `reasoningContentField: "reasoning_content"`,
+  `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction rides [[frank-zealot]].
 - Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
  pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
  Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
@@ -124,14 +128,18 @@ created it — earlier topologies in .
  prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
  85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
  re-measure the warm ratio after any pool change.
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
-  155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
-  its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
-  3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
-  over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
-  target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
-  is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
-  under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
+- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling, because the harness
+  clamps `max_tokens` to `contextWindow − est − 4096`, its chars/4 `est` UNDER-counts code/agent
+  traffic by ~3 %, and the compaction trigger fires at 95 % of the window (`config.yml
+  thresholdPercent`, while the comments designed around 85 % — check the trigger BEFORE touching a
+  window; that disagreement is why long sessions kept drawing 400s). The 3 % is measured, not
+  theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 — over by 354, a 400.
+  At the live pair (ceiling 262,144 / window 200,000) the worst-case ordinary request is
+  200,000 − 4,096 + 3 % ≈ 201.8k, and the trigger point is 190,000 est × 1.03 + 45,056 ≈ 240.7k —
+  both inside the ceiling with ~60k of margin. Compaction rides [[frank-zealot]] at
+  ≈ threshold − keepRecent(20k) ≈ 175k measured against its 200,704-token slot. The seam in
+  `tests/frank_dragoon_unit.bats` re-runs this arithmetic at the 95 % trigger and reads the window
+  from THIS page, so ceiling/window pairs that drift back together fail the suite, not a session.
 - **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
  503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
  together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
@@ -248,9 +256,14 @@ lane is **armed and serving** since 2026-10-19 (`HALOGEN_NPU_MODELS=qwen3-embedd
   (`dankclaw/udev/98-halogen-npu-access.rules`, installed and re-triggered by the deployer, `0666` like the GPU
   compute nodes beside it); the serve script refuses any node that is not other-accessible, so the fault cannot
   return as a crash-loop.
-2. The host's XRT reachable inside: there is **no `/opt/xilinx/xrt`** here — distro packages put the three files the
-   container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`, `libxrt_driver_xdna.so.2`) in
-   `/usr/lib/x86_64-linux-gnu`, where all three exist.
+2. The host's XRT reachable inside, at **THREE prefixes per library**: there is **no `/opt/xilinx/xrt`** here —
+   distro packages put the three files the container needs (`libxrt_coreutil.so.2`, `libxrt_core.so.2`,
+   `libxrt_driver_xdna.so.2`) in `/usr/lib/x86_64-linux-gnu`. The serve script mounts each REAL file (never a
+   symlink) at `/opt/xilinx/xrt/lib/`, at its own path, AND at `/opt/xilinx/lib/<multiarch>/` — the NPU child
+   searches that third prefix and a missing one is fatal: since the host's `libxrt2` was reconfigured
+   2026-10-20 the child died with `No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'` and the
+   engine supervisor took the WHOLE container down on it (88-restart loop, chat dead beside a healthy GPU line).
+   A dead lane is not survivable here — arm the lane only with all three prefixes mounted.
 3. **The GPU fabric clock held at its top level, or the engine refuses to start.** GPU and NPU work share the die:
   while `pp_dpm_fclk` is changing speed, concurrent GPU work can hang the machine and corrupt NPU results. The
   engine holds the clock itself only when it runs as root with `/sys` mounted; `frank-dragoon` is a **rootless**
@@ -17,6 +17,9 @@ logging {
 // Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
 // host network, so it pushes by loopback, never by container name — a root
 // container is not on the rootless dankclaw_default network.
+// GRAFANA_HOST_LABEL is replaced with this host's name by
+// copy_alloy_config at deploy time; it must not appear verbatim in the
+// installed config.
 loki.write "default" {
  endpoint {
    url = "http://127.0.0.1:3100/loki/api/v1/push"
@@ -32,7 +35,34 @@ loki.write "default" {
 loki.source.journal "journal" {
  format_as_json = true
  labels = {
-    job = "systemd-journal",
+    job  = "systemd-journal",
+    host = "GRAFANA_HOST_LABEL",
+  }
+  forward_to = [loki.process.relabel.receiver]
+}
+
+// Promote the fields that ARE the point of the log stream to real Loki labels.
+// Without this the journal arrives as one opaque JSON line per entry and every
+// per-container view is structurally impossible: Loki metrics queries can only
+// group by INDEXED stream labels, so `sum by (container_name)` returns a single
+// anonymous series and the logs-overview board read empty beside 6.5k
+// lines/5min of live traffic (measured 2026-10-21). CONTAINER_NAME is what
+// podman's journald logging writes (uppercase); host units lack it and stay on
+// the host-only stream. `format_as_json = true` above guarantees every line
+// reaching this stage is a JSON object, so there is no non-JSON branch to
+// protect — alloy 1.21's json stage has no `on_error` attribute (validated
+// against the pinned image); a malformed line would drop, which is the
+// correct fate for a line Loki could not index anyway.
+loki.process "relabel" {
+  stage.json {
+    expressions = {
+      container = "CONTAINER_NAME",
+    }
+  }
+  stage.labels {
+    values = {
+      container = "container",
+    }
  }
  forward_to = [loki.write.default.receiver]
 }
@@ -0,0 +1,18 @@
+# Blackbox exporter modules — the honest answer to "is this HTTP service
+# answering?". A Prometheus scrape cannot express it (an HTML front page is a
+# 200 that fails the scrape protocol), which is why the old `services` and
+# `comfyui` jobs sat permanently red on `unsupported Content-Type`.
+# See serverAppConfigs-docs/Architecture/Stale Signal Audit.md §1.
+# Loopback bind: the image's default 0.0.0.0:9115 would ride the LAN on this
+# Network=host container. Prometheus (bridge) still reaches it — its traffic
+# arrives on the podman gateway interface, which is not loopback.
+network:
+  listen_address: 127.0.0.1:9115
+
+modules:
+  http_2xx:
+    prober: http
+    timeout: 10s
+    http:
+      preferred_ip_protocol: ip4
+      valid_status_codes: [] # default: 2xx + 3xx
@@ -0,0 +1,137 @@
+{
+  "uid": "containers",
+  "title": "Containers — Dankclaw & Dankserver",
+  "description": "Per-container CPU and memory for the quadlet app stacks, from cadvisor (a rootless quadlet on each host). cadvisor keys each series by the cgroup path, and Prometheus relabels the quadlet unit name (app.slice/<name>.service) into a `container` label. Panels use only the metric families cadvisor actually exports for these rootless containers: CPU usage and memory working-set/limit/OOM. Network and filesystem are NOT exported for them (no netns/classid attribution under the rootless+host-PID layout), so no panels pretend otherwise. See serverAppConfigs-docs/Infrastructure/Observability.md.",
+  "tags": ["fleet", "cadvisor", "containers"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "host",
+        "label": "Host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\"}, host)", "refId": "host_var" },
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      },
+      {
+        "name": "container",
+        "label": "Container",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\"}, container)", "refId": "container_var" },
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "row",
+      "title": "CPU",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+      "panels": []
+    },
+    {
+      "id": 2,
+      "type": "timeseries",
+      "title": "CPU usage (cores)",
+      "description": "CPU-seconds/s per container = cores busy. A steady high line is a busy service; a spike is a burst (a render, an index, a backup). Only containers are shown, not the host or its daemons.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 1 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "sum by (host, container) (rate(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m]))",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "bargauge",
+      "title": "Top containers by CPU (cores)",
+      "description": "Instant ranking over the selected window. Tells you which container to look at when the host is loaded.",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 1 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "max": null, "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 4 } ] } }, "overrides": [] },
+      "options": { "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false }, "displayMode": "gradient", "orientation": "horizontal", "showUnfilled": true },
+      "targets": [
+        {
+          "expr": "topk(12, sum by (host, container) (rate(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m])))",
+          "legendFormat": "{{host}} / {{container}}",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "row",
+      "title": "Memory",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 10 },
+      "panels": []
+    },
+    {
+      "id": 5,
+      "type": "timeseries",
+      "title": "Memory working set",
+      "description": "container_memory_working_set_bytes per container — the number the OOM killer acts on. Cache-inclusive and reclaimable; a flat high line is resident data, a sawtooth is churn.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 11 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "timeseries",
+      "title": "Memory vs host total (%)",
+      "description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "type": "timeseries",
+      "title": "OOM kills",
+      "description": "container_oom_events_total rate. Any non-zero line means the kernel killed a process inside that container — the honest signal that its memory limit is too tight for its real working set.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 20, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "sum by (host, container) (rate(container_oom_events_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m]))",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}
@@ -18,6 +18,16 @@
        "includeAll": true,
        "multi": true,
        "current": { "text": "All", "value": "$__all" }
+      },
+      {
+        "name": "gpuhost",
+        "label": "gpu host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(gpu_utilization_percent, host)", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
      }
    ]
  },
@@ -44,7 +54,7 @@
      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
-      "targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
+      "targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
    },
    {
      "id": 4, "type": "stat", "title": "Open file handles",
@@ -139,6 +149,56 @@
        { "expr": "sum by (job,device) (rate(node_network_receive_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} RX err", "refId": "A" },
        { "expr": "sum by (job,device) (rate(node_network_transmit_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} TX err", "refId": "B" }
      ]
+    },
+    {
+      "id": 13, "type": "timeseries", "title": "GPU busy %",
+      "description": "Per-GPU utilization from gpu-exporter (amdgpu gpu_busy_percent / nvidia-smi). The picker above selects the host; each series is one card.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 36 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_utilization_percent{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 14, "type": "timeseries", "title": "GPU VRAM used",
+      "description": "Device memory in use. On the R9700s (frank-archon) and the 3090 (frank-zealot) this is the loaded model's footprint.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 36 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_memory_used_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 15, "type": "timeseries", "title": "GPU VRAM used %",
+      "description": "Used / total device memory. Near 100% is the pre-OOM wall.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 44 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100 }, "overrides": [] },
+      "targets": [
+        { "expr": "100 * gpu_memory_used_bytes{host=~\"$gpuhost\"} / gpu_memory_total_bytes{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 16, "type": "timeseries", "title": "GPU temperature",
+      "description": "Package temperature. AMD edge temp (hwmon temp1_input); NVIDIA gpu temp.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 44 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "celsius", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_temperature_celsius{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
+    },
+    {
+      "id": 17, "type": "timeseries", "title": "GPU power draw",
+      "description": "Board power draw. Emitted only where readable: NVIDIA and the Dankclaw iGPU. The R9700s have no sysfs power sensor, so they are simply absent here — never a fabricated 0.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 52 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "watt", "min": 0 }, "overrides": [] },
+      "targets": [
+        { "expr": "gpu_power_watts{host=~\"$gpuhost\"}", "legendFormat": "{{host}} {{gpu}} ({{model}})", "refId": "A" }
+      ]
    }
  ]
 }
@@ -194,7 +194,7 @@
      "id": 6,
      "type": "timeseries",
      "title": "Firing fleet alerts",
-      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
+      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
      "gridPos": {
        "h": 8,
        "w": 12,
@@ -2,74 +2,242 @@
  "uid": "logs-overview",
  "title": "Logs \u2014 Loki (systemd journal)",
  "description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
-  "tags": ["logs", "loki"],
+  "tags": [
+    "logs",
+    "loki"
+  ],
  "timezone": "browser",
  "schemaVersion": 39,
  "refresh": "30s",
-  "time": { "from": "now-1h", "to": "now" },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
  "templating": {
    "list": [
      {
        "name": "container",
        "label": "container",
        "type": "query",
-        "datasource": { "type": "loki", "uid": "loki" },
-        "query": { "query": "{job=\"systemd-journal\"} | json", "label": "container_name", "refId": "A" },
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "query": {
+          "query": "label_values({job=\"systemd-journal\"}, container)",
+          "refId": "LokiVariableQueryEditor-VariableQuery"
+        },
        "includeAll": true,
        "multi": true,
        "refresh": 2,
-        "current": { "text": "All", "value": "$__all" }
+        "current": {
+          "text": "All",
+          "value": "$__all"
+        }
      }
    ]
  },
  "panels": [
    {
-      "id": 1, "type": "timeseries", "title": "Log volume by container (lines/s)",
-      "description": "Parse the journal JSON, count lines per container. Spikes = a chatty or crashing unit.",
-      "gridPos": { "h": 9, "w": 24, "x": 0, "y": 0 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps", "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] },
-      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+      "id": 1,
+      "type": "timeseries",
+      "title": "Log volume by container (lines/s)",
+      "description": "Per-container journal line rate. The `container` label is promoted at ingestion by Alloy (loki.process) from podman's CONTAINER_NAME journald field - Loki metrics can only group by indexed stream labels.",
+      "gridPos": {
+        "h": 9,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "custom": {
+            "fillOpacity": 20,
+            "stacking": {
+              "mode": "normal"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 2, "type": "timeseries", "title": "Total log rate",
+      "id": 2,
+      "type": "timeseries",
+      "title": "Total log rate",
      "description": "All journal lines per second.",
-      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 9 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
-      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))", "legendFormat": "all lines/s", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))",
+          "legendFormat": "all lines/s",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 3, "type": "timeseries", "title": "Error-level line rate",
+      "id": 3,
+      "type": "timeseries",
+      "title": "Error-ish line rate (text match)",
      "description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
-      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 9 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "fixed", "fixedColor": "red" } }, "overrides": [] },
-      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))", "legendFormat": "error lines/s", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "color": {
+            "mode": "fixed",
+            "fixedColor": "red"
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))",
+          "legendFormat": "error lines/s",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 4, "type": "timeseries", "title": "Errors by container",
+      "id": 4,
+      "type": "timeseries",
+      "title": "Errors by container",
      "description": "Which unit is throwing. Rank by the top offenders.",
-      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 17 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
-      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} |~ \"(?i)(error|panic|fatal)\" [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 5, "type": "stat", "title": "Error lines (last 15m)",
+      "id": 5,
+      "type": "stat",
+      "title": "Error lines (last 15m)",
      "description": "Absolute count over the window, not a rate.",
-      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 17 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 50 }, { "color": "red", "value": 500 }] } }, "overrides": [] },
-      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))", "instant": true, "legendFormat": "errors/15m", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 50
+              },
+              {
+                "color": "red",
+                "value": 500
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))",
+          "instant": true,
+          "legendFormat": "errors/15m",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 6, "type": "logs", "title": "Journal (filtered)",
+      "id": 6,
+      "type": "logs",
+      "title": "Journal (filtered)",
      "description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
-      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 25 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
-      "targets": [{ "expr": "{job=\"systemd-journal\"} | json | container_name=~\"$container\"", "refId": "A" }]
+      "gridPos": {
+        "h": 12,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "options": {
+        "showTime": true,
+        "wrapLogMessage": true,
+        "sortOrder": "Descending",
+        "enableLogDetails": true
+      },
+      "targets": [
+        {
+          "expr": "{job=\"systemd-journal\", container=~\"$container\"}",
+          "refId": "A"
+        }
+      ]
    }
  ]
-}
+}
@@ -2,95 +2,384 @@
  "uid": "observability",
  "title": "Observability \u2014 self-monitoring",
  "description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
-  "tags": ["observability", "meta"],
+  "tags": [
+    "observability",
+    "meta"
+  ],
  "timezone": "browser",
  "schemaVersion": 39,
  "refresh": "30s",
-  "time": { "from": "now-6h", "to": "now" },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
  "panels": [
    {
-      "id": 1, "type": "stat", "title": "Targets up",
-      "gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] } }, "overrides": [] },
-      "targets": [{ "expr": "count(up == 1)", "instant": true, "refId": "A" }]
+      "id": 1,
+      "type": "stat",
+      "title": "Targets up",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 1)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 2, "type": "stat", "title": "Targets down",
-      "description": "Known structural downs: comfyui (no metrics endpoint), gitea (metrics off), and the homepage 'services' probes. Real regressions ride up from here too \u2014 check the table before dismissing.",
-      "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 1 }] } }, "overrides": [] },
-      "targets": [{ "expr": "count(up == 0)", "instant": true, "refId": "A" }]
+      "id": 2,
+      "type": "stat",
+      "title": "Targets down",
+      "description": "Every scrape target that is down. ZERO IS THE HEALTHY STATE: the HTML-probe jobs and the disabled Gitea metrics were retired 2026-10-21 (HTTP liveness now rides the blackbox exporter, job \"services\"). A non-zero count here is a real regression - read /api/v1/targets lastError before triaging.",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 4,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 0)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 3, "type": "stat", "title": "TSDB head series",
+      "id": 3,
+      "type": "stat",
+      "title": "TSDB head series",
      "description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
-      "gridPos": { "h": 4, "w": 5, "x": 8, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
-      "targets": [{ "expr": "prometheus_tsdb_head_series", "instant": true, "refId": "A" }]
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 8,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_tsdb_head_series",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 4, "type": "stat", "title": "Discovered targets",
-      "gridPos": { "h": 4, "w": 5, "x": 13, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
-      "targets": [{ "expr": "sum(prometheus_sd_discovered_targets)", "instant": true, "refId": "A" }]
+      "id": 4,
+      "type": "stat",
+      "title": "Discovered targets",
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 13,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(prometheus_sd_discovered_targets)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 5, "type": "stat", "title": "Prometheus mem",
-      "gridPos": { "h": 4, "w": 3, "x": 18, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
-      "targets": [{ "expr": "process_resident_memory_bytes{job=\"prometheus\"}", "instant": true, "refId": "A" }]
+      "id": 5,
+      "type": "stat",
+      "title": "Prometheus mem",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "process_resident_memory_bytes{job=\"prometheus\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 6, "type": "stat", "title": "Grafana up",
-      "gridPos": { "h": 4, "w": 3, "x": 21, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "mappings": [{ "type": "value", "options": { "1": { "text": "UP", "color": "green" }, "0": { "text": "DOWN", "color": "red" } } }], "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] } }, "overrides": [] },
-      "targets": [{ "expr": "up{job=\"grafana\"}", "instant": true, "refId": "A" }]
+      "id": 6,
+      "type": "stat",
+      "title": "Grafana up",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            {
+              "type": "value",
+              "options": {
+                "1": {
+                  "text": "UP",
+                  "color": "green"
+                },
+                "0": {
+                  "text": "DOWN",
+                  "color": "red"
+                }
+              }
+            }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "up{job=\"grafana\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 7, "type": "table", "title": "Target health",
+      "id": 7,
+      "type": "table",
+      "title": "Target health",
      "description": "Every scrape target and its current up. Sort by Value to surface regressions.",
-      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 4 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "options": { "sortBy": [{ "displayName": "Value", "sortOrder": "Asc" }] },
-      "targets": [{ "expr": "up", "format": "table", "instant": true, "refId": "A" }]
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "options": {
+        "sortBy": [
+          {
+            "displayName": "Value",
+            "sortOrder": "Asc"
+          }
+        ]
+      },
+      "targets": [
+        {
+          "expr": "up",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 8, "type": "timeseries", "title": "Scrape interval drift (p99 actual)",
+      "id": 8,
+      "type": "timeseries",
+      "title": "Scrape interval drift (p99 actual)",
      "description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
-      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 4 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
-      "targets": [{ "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}", "legendFormat": "{{instance}}", "refId": "A" }]
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 9, "type": "timeseries", "title": "Scrape samples/s",
+      "id": 9,
+      "type": "timeseries",
+      "title": "Scrape samples/s",
      "description": "Total samples ingested across all jobs.",
-      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
-      "targets": [{ "expr": "sum(rate(scrape_samples_scraped[5m]))", "legendFormat": "samples/s", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(rate(scrape_samples_scraped[5m]))",
+          "legendFormat": "samples/s",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 10, "type": "timeseries", "title": "Loki request rate",
+      "id": 10,
+      "type": "timeseries",
+      "title": "Loki request rate",
      "description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
-      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
-      "targets": [{ "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))", "legendFormat": "{{route}}", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))",
+          "legendFormat": "{{route}}",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 11, "type": "timeseries", "title": "Exporter process CPU",
+      "id": 11,
+      "type": "timeseries",
+      "title": "Exporter process CPU",
      "description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
-      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
-      "targets": [{ "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))", "legendFormat": "{{job}}", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 21
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))",
+          "legendFormat": "{{job}}",
+          "refId": "A"
+        }
+      ]
    }
  ]
-}
+}
@@ -60,20 +60,31 @@ scrape_configs:
      - targets: ["loki:3100"]
    metrics_path: "/metrics"

-  # ComfyUI (no native /metrics; probe the UI)
-  - job_name: "comfyui"
-    metrics_path: "/"
-    static_configs:
-      - targets: ["comfyui:8188"]
-
-  # Blackbox-style probes for service up/down (published LAN ports;
-  # gitea and comfyui are covered by their own jobs above)
+  # HTTP liveness, probed honestly through the blackbox exporter. Scraping a
+  # web root is NOT a health check — an HTML front page fails the scrape
+  # protocol (`unsupported Content-Type`), which is exactly why the old
+  # `services` and `comfyui` jobs were red forever beside healthy services
+  # (Stale Signal Audit §1). The prober is a host-netns container reached at
+  # the podman gateway name (HostAddr= on the prometheus unit); it in turn
+  # reaches both the host LAN IP and bridge container names. comfyui's UI probe
+  # lives here now — its own metrics-scrape job is retired, one owner per
+  # service.
  - job_name: "services"
-    metrics_path: "/"
+    metrics_path: /probe
+    params:
+      module: [http_2xx]
    static_configs:
      - targets:
-          - "192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
-          - "192.168.0.143:3005"   # OpenWebUI
+          - "http://192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
+          - "http://192.168.0.143:3005"   # OpenWebUI
+          - "http://comfyui:8188"         # ComfyUI UI (gfx1151 generator)
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: "host.containers.internal:9115"

  # LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
  # this container, not the guarded vhosts: these are machine clients and
@@ -131,3 +142,88 @@ scrape_configs:
      - targets: ["192.168.0.235:7770"]
        labels:
          service: frank-archon
+
+  # Per-container metrics from cadvisor (a rootless quadlet on each host).
+  # cadvisor keys every series by the cgroup path; for a quadlet container that
+  # path is …/app.slice/<name>.service[/libpod-payload-<hex>], so the unit NAME
+  # lives in it. The relabel below pulls that name out into a `container` label
+  # and keeps only the per-unit aggregate — cadvisor's raw host root,
+  # system.slice daemons and machine.slice scopes carry no such name and are
+  # dropped, as is the per-CPU-core split (only cpu="total" survives), so the
+  # scrape stays small. cadvisor cannot bind per-interface, so it is reached at
+  # the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
+  # DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
+  # connection whether the port is a host-net bind or a published port (the
+  # original note here claimed published ports were exempt; measured 2026-10-21
+  # from inside this container: LAN refused, gateway answers). Cross-host
+  # targets (dankserver) have no hairpin problem and stay plain LAN.
+  - job_name: "cadvisor-dankclaw"
+    static_configs:
+      - targets: ["host.containers.internal:8080"]
+        labels:
+          host: dankclaw
+    metric_relabel_configs:
+      - source_labels: [id]
+        regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
+        target_label: container
+        replacement: '$1'
+      # Keep only the unit-level cgroup; drop its payload/control children so a
+      # container is not counted twice.
+      - source_labels: [id]
+        regex: '.*/\.service/.*'
+        action: drop
+      # Drop everything that is not a named unit (no `container` was extracted).
+      - source_labels: [container]
+        regex: '.+'
+        action: keep
+      # cpu="total" for cpu series; metrics with no cpu label (memory, network,
+      # filesystem) are kept.
+      - source_labels: [cpu]
+        regex: '^$|total'
+        action: keep
+      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
+        action: labeldrop
+
+  - job_name: "cadvisor-dankserver"
+    static_configs:
+      - targets: ["192.168.0.235:8080"]
+        labels:
+          host: dankserver
+    metric_relabel_configs:
+      - source_labels: [id]
+        regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
+        target_label: container
+        replacement: '$1'
+      - source_labels: [id]
+        regex: '.*/\.service/.*'
+        action: drop
+      - source_labels: [container]
+        regex: '.+'
+        action: keep
+      - source_labels: [cpu]
+        regex: '^$|total'
+        action: keep
+      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
+        action: labeldrop
+
+  # Per-GPU metrics (gpu_utilization_percent / gpu_memory_used_bytes /
+  # gpu_memory_total_bytes / gpu_temperature_celsius / gpu_power_watts) from the
+  # host-process gpu-exporter (scripts/gpu-exporter.py). One job per host, same
+  # reach rules as node/cadvisor: Dankclaw is scraped at the podman gateway
+  # because the netavark hairpin quirk makes Prometheus fail to dial its own
+  # host LAN IP; Dankserver is plain LAN. The gpu label (PCI BDF on AMD, index on
+  # NVIDIA) + vendor/model come from the exporter, so the GPU row on the
+  # host-system dashboard filters by host. Power is emitted only where readable
+  # (NVIDIA + the Dankclaw iGPU); the R9700s have no sysfs power, so they simply
+  # have no gpu_power_watts series — no fabricated zero.
+  - job_name: "gpu-dankclaw"
+    static_configs:
+      - targets: ["host.containers.internal:9401"]
+        labels:
+          host: dankclaw
+
+  - job_name: "gpu-dankserver"
+    static_configs:
+      - targets: ["192.168.0.235:9401"]
+        labels:
+          host: dankserver
@@ -193,7 +193,18 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
 }

@test "prometheus targets are host:port (no URL schemes)" {
-    ! grep -nE '^\s*-\s*"?http' "$REPO_ROOT/shared/prometheus/prometheus.yml"
+    # Plain scrape targets are host:port; a URL target is only meaningful for
+    # the blackbox prober job, whose relabel_configs copy it to __param_target
+    # and point __address__ at the prober. Anywhere else a URL target is the
+    # old HTML-probe defect wearing a new hat.
+    awk '
+        /job_name:/ { job=$0; sub(/.*"/, "", job); sub(/".*/, "", job); probe = 0 }
+        /metrics_path: \/probe/ { probe = 1 }
+        /^[[:space:]]*-[[:space:]]*"http/ && !probe {
+            print FILENAME ":" FNR ": URL target outside a prober job: " $0; bad = 1
+        }
+        END { exit bad }
+    ' "$REPO_ROOT/shared/prometheus/prometheus.yml"
 }

@test "observability containers run as container root (rootless = host user)" {
@@ -125,3 +125,28 @@ load helpers
    [ ! -e "$sandbox/frank-default-sightsound" ]
    [ -e "$BATS_TEST_TMPDIR/.AGENTTRASH/$(date +%Y%m%d)-retired-deploy-state/frank-default-sightsound" ]
 }
+
+@test "the client and remote allowlists agree per target" {
+    # Two gates guard `deploy.sh --service X`: the client's valid_service and
+    # the remote runner's service_is_allowed. When only one side gained a
+    # service the other answered "is not deployed on <target>" and the only
+    # repair path was a fleet-wide drain — cadvisor and gpu-exporter both fell
+    # through this crack within a day of each other. The lists MUST agree;
+    # searxng-network is the one sanctioned exception (client-only network
+    # unit, no remote dispatch handler).
+    run env OUT="$BATS_TEST_TMPDIR" bash -c '
+        grep -oE "dank(claw|server):[a-z0-9-]+" \
+            <(sed -n "/^valid_service()/,/^}/p" "$1/scripts/deploy.sh") \
+            | grep -v "dankclaw:searxng-network" | sort -u > "$OUT/cli"
+        grep -oE "dank(claw|server):[a-z0-9-]+" \
+            <(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
+            | sort -u > "$OUT/rem"
+        diff "$OUT/cli" "$OUT/rem"
+    ' _ "$REPO_ROOT"
+    [ "$status" -eq 0 ] || { echo "deploy.sh vs deploy-remote.sh allowlist drift:"; echo "$output"; return 1; }
+    # Drift aside, the gate must actually know the host-process exporters —
+    # their first-ever deploy has to pass the client gate.
+    for pair in dankclaw:cadvisor dankclaw:gpu-exporter dankserver:cadvisor dankserver:gpu-exporter; do
+        grep -q "$pair" "$BATS_TEST_TMPDIR/cli" || { echo "client gate missing $pair"; return 1; }
+    done
+}
@@ -3,7 +3,7 @@
 # of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the
 # Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
 # digest-pinned), driven by the root unit's serve script — 2 slots over ONE
-# 200,704-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0)
+# 401,408-position KV pool at the native 262,144 ceiling, MTP head always on (HALOGEN_SPEC_ADAPT=0)
 # and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune
 # shape, which itself cut over the halogen 27B-dense shape on 2026-10-06.
 #
@@ -85,7 +85,7 @@ good_env() {
        "FRANK_DRAGOON_VISION_BYTES=0"
        "FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
        "HALOGEN_MODEL_ID=frank-dragoon"
-        "HALOGEN_CTX=200704"
+        "HALOGEN_CTX=262144"
        "HALOGEN_KV_POOL_POSITIONS=401408"
        "HALOGEN_KV_SLOTS=2"
        "HALOGEN_SPEC_ADAPT=0"
@@ -159,9 +159,11 @@ good_env() {
    reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
    # The pool is ONE budget behind the slots (boot ledger form: "2 slots over
    # ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
-    # a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx
-    # (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be
-    # resident at once — frank-zealot parity on total in-flight context. What
+    # a per-slot private pool. Since 2026-10-21 the ceiling is the checkpoint's
+    # native 262,144 while the pool stays 401,408 (raised 2026-10-18): two
+    # MAXIMUM harness-length (200,000-token) conversations are resident at once;
+    # two literal ceiling-length requests would queue — widening that is a pool
+    # raise, not a window move.
    # still binds: the pool must back ONE full-length request (below that the
    # advertised per-request ceiling would be unreachable, hence pool >= ctx),
    # it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
@@ -287,7 +289,7 @@ $(cat "$TEST_TMP/serve.log")"
    [[ "$output" == *"--name frank-dragoon-halogen"* ]]
    [[ "$output" == *"--publish 7770:8731"* ]]
    [[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
-    [[ "$output" == *"-e HALOGEN_CTX=200704"* ]]
+    [[ "$output" == *"-e HALOGEN_CTX=262144"* ]]
    [[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
    [[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
    [[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
@@ -306,27 +308,32 @@ $(cat "$TEST_TMP/serve.log")"

@test "the shipped env keeps the harness fit law inside the ceiling" {
    # HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400
-    # naming the limit, never a silent truncation). The law is INVARIANT to
-    # the value: the harness window must sit ~45k UNDER it, because the
-    # harness clamps max_tokens = min(45056, window − est − 4096), compacts
-    # at 85 % of its declared window, and its chars/4 est UNDER-counts the
-    # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens at
-    # est 152,938 on 2026-10-07 — with window == ceiling, every request over
-    # ~65 % of the window was a guaranteed 400 before compaction could fire).
-    # The shipped pair is ctx 200704 / window 155648 (lowered from
-    # 262144/217088 on 2026-10-17); the gap must absorb the error at BOTH
-    # monotone maxima of est + budget (est ≤ window − 4096):
-    #   full budget at the compaction threshold: 1.03*132300 + 45056
-    #   clamped budget at the clamp's zero point: 1.03*151552 + 0
-    # and the compaction request that rides on frank-zealot (its
-    # 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k).
-    grep -qE '^HALOGEN_CTX=200704$' "$ENV_FILE"
-    ctx=200704
-    hw=155648; mt=45056
-    thr=$(( hw * 85 / 100 ))
+    # naming the limit, never a silent truncation). The law: the harness
+    # window must sit ~45k UNDER the ceiling, because the harness clamps
+    # max_tokens = min(45056, window − est − 4096), compacts at the trigger
+    # percent of its declared window, and its chars/4 est UNDER-counts the
+    # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens
+    # at est 152,938 on 2026-10-07 — with window ≈ ceiling, every long
+    # request is a guaranteed 400 before compaction can fire). The 2026-10-21
+    # pair: ceiling 262144 (native) under a harness window of 200,000.
+    # The declared window is read from the endpoint page — the one owner —
+    # so env, models.yml and page that disagree FAIL HERE, not as a 400
+    # mid-session (the recurring failure this seam exists for).
+    grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE"
+    ctx="$(sed -nE 's/^HALOGEN_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
+    page="$ROOT/serverAppConfigs-docs/LLM/frank-dragoon.md"
+    grep -qE 'contextWindow: 200000' "$page"
+    hw="$(grep -oE 'contextWindow: [0-9]+' "$page" | head -1 | grep -oE '[0-9]+')"
+    mt=45056
+    # Invariant checked at the live 95 % trigger (config.yml
+    # thresholdPercent) — strictly stronger than the 85 % design point.
+    # Both monotone maxima of est + budget (est ≤ window − 4096):
+    thr=$(( hw * 95 / 100 ))
    [ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
    zero=$(( hw - 4096 ))
    [ $(( zero * 103 / 100 )) -le "$ctx" ]
+    # The compaction request rides on frank-zealot's 200,704-token slot:
+    # input ≈ threshold − keepRecent(20k).
    [ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
 }

@@ -423,9 +430,12 @@ npu_fixture() {
    # the stock root:render 0660 mode, and the serve gate checks exactly that.
    touch "$TEST_TMP/accel0"
    chmod 666 "$TEST_TMP/accel0"
-    mkdir -p "$TEST_TMP/xrt"
+    # basename carries a multiarch shape: the third mount prefix is derived
+    # from the XRT dir's basename (the NPU child searches
+    # /opt/xilinx/lib/<multiarch>/, measured 2026-10-21).
+    mkdir -p "$TEST_TMP/x86_64-linux-gnu"
    for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
-        touch "$TEST_TMP/xrt/$l"
+        touch "$TEST_TMP/x86_64-linux-gnu/$l"
    done
    mkdir -p "$TEST_TMP/npu/qwen3-embedding-0.6b/devices"
    mkdir -p "$TEST_TMP/sys/class/drm/card1/device"
@@ -435,7 +445,7 @@ npu_fixture() {
        "HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"
        "FRANK_DRAGOON_NPU_DIR=$TEST_TMP/npu"
        "FRANK_DRAGOON_DEV_ACCEL=$TEST_TMP/accel0"
-        "FRANK_DRAGOON_XRT_DIR=$TEST_TMP/xrt"
+        "FRANK_DRAGOON_XRT_DIR=$TEST_TMP/x86_64-linux-gnu"
        "FRANK_DRAGOON_SYSFS=$TEST_TMP/sys"
    )
 }
@@ -465,11 +475,15 @@ $(cat "$TEST_TMP/serve.log")"
    ! grep -q 'accel0' "$TEST_TMP/serve.log"
 }

-@test "the lane arms the accel node, the staged model dir and each XRT library twice" {
-    # Each library mounts at the image's XRT path AND at its own path: the
-    # distro libraries find each other where they were installed, and an
-    # /opt/xilinx/xrt of symlinks into the system dir dangles inside the
-    # container (the vendor calls that out by name).
+@test "the lane arms the accel node, the staged model dir and each XRT library three times" {
+    # Each library mounts at the image's XRT path, at its own path (the distro
+    # libraries find each other where they were installed), and at
+    # /opt/xilinx/lib/<multiarch>/ — the prefix the NPU child searches and
+    # names when it is missing (`No such library
+    # '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'`, measured
+    # 2026-10-21: the lane died and the supervisor took the whole container
+    # with it, 88-restart loop). An /opt/xilinx/xrt of symlinks into the
+    # system dir dangles inside the container (the vendor calls that out).
    podman_stub
    good_env
    npu_fixture
@@ -484,8 +498,9 @@ $(cat "$TEST_TMP/serve.log")"
    [[ "$output" != *":/models/npu"* ]]
    [[ "$output" == *"-e HALOGEN_NPU_MODELS=qwen3-embedding-0.6b"* ]]
    for l in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
-        [[ "$output" == *"-v $TEST_TMP/xrt/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
-        [[ "$output" == *"-v $TEST_TMP/xrt/$l:$TEST_TMP/xrt/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/xrt/lib/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:$TEST_TMP/x86_64-linux-gnu/$l:ro"* ]]
+        [[ "$output" == *"-v $TEST_TMP/x86_64-linux-gnu/$l:/opt/xilinx/lib/x86_64-linux-gnu/$l:ro"* ]]
    done
    # The Flash payload still rides the same container: the lane is additive.
    [[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
@@ -560,7 +575,7 @@ $(cat "$TEST_TMP/serve.log")"
    podman_stub
    good_env
    npu_fixture
-    rm "$TEST_TMP/xrt/libxrt_driver_xdna.so.2"
+    rm "$TEST_TMP/x86_64-linux-gnu/libxrt_driver_xdna.so.2"
    run_with_log
    [ "$status" -ne 0 ]
    [[ "$output" == *"libxrt_driver_xdna.so.2 missing"* ]]
@@ -0,0 +1,227 @@
+#!/usr/bin/env bats
+# Seam: the GPU metrics exporter's black-box contract.
+#
+# The exporter reads amdgpu sysfs (one card per /sys/class/drm/cardN that has a
+# device/gpu_busy_percent file) and, on the box that has a working NVIDIA stack,
+# one `nvidia-smi --query-gpu` CSV, and emits the unified `gpu_*` Prometheus
+# families the host-system dashboard graphs. Both sources are injectable so the
+# whole thing is testable with fixtures on a GPU-less dev box.
+#
+# Only the inputs a consumer sees are pinned: the metric families, their label
+# set, and that absent hardware yields ABSENT series (never a fabricated 0 — a
+# wrong number on a dashboard gets believed). The R9700s expose no sysfs
+# power1_average, so those cards legitimately carry no gpu_power_watts line.
+
+load helpers
+
+EXPORTER="$REPO_ROOT/scripts/gpu-exporter.py"
+
+# Build a fixture amdgpu sysfs tree. $BATS_TEST_TMPDIR/sys
+mk_amd() { # gpu busy vram_used vram_total power_uw_or_blank temp_mc
+    local card=$1 busy=$2 used=$3 total=$4 power=$5 temp=$6
+    local d="$SYS/card$card/device"
+    mkdir -p "$d/hwmon/hwmon9"
+    printf '%s\n' "$busy" > "$d/gpu_busy_percent"
+    printf '%s\n' "$used" > "$d/mem_info_vram_used"
+    printf '%s\n' "$total" > "$d/mem_info_vram_total"
+    printf '%s\n' "$power" > "$d/power1_average"          # blank => absent on R9700
+    [ -n "$temp" ] && printf '%s\n' "$temp" > "$d/hwmon/hwmon9/temp1_input"
+    printf 'DRIVER=amdgpu\nPCI_ID=1002:7551\nPCI_SLOT_NAME=0000:03:00.0\n' > "$d/uevent"
+}
+
+setup() {
+    SYS="$BATS_TEST_TMPDIR/sys"
+    mkdir -p "$SYS"
+}
+
+@test "exporter exists and compiles" {
+    [ -f "$EXPORTER" ]
+    python3 -m py_compile "$EXPORTER"
+}
+
+@test "amd card with full sysfs yields util/vram/temp/power with the right labels and units" {
+    mk_amd 1 42 1073741824 34359738368 250000000 71000   # 250 W, 71 C
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    # util is a raw percent, VRAM verbatim bytes, temp milli->C, power microW->W.
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="amd"[^}]*\} 42$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*\} 1073741824$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_memory_total_bytes\{[^}]*\} 34359738368$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_temperature_celsius\{[^}]*\} 71$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_power_watts\{[^}]*\} 250$' || { echo "$output"; return 1; }
+    # identity labels: the PCI slot is a stable unique id, vendor marks the family.
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{.*gpu="0000:03:00\.0"' || { echo "$output"; return 1; }
+}
+
+@test "card with an empty power file and no hwmon yields util/vram but omits power and temp" {
+    mk_amd 1 99 34108338176 34208743424 "" ""           # the R9700 shape
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*\} 99$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*\} 34108338176$'
+    # Absent means ABSENT, not a zero the dashboard would plot as a real reading.
+    ! echo "$output" | grep -Eq '^gpu_power_watts\{' || { echo "fabricated power line: $output"; return 1; }
+    ! echo "$output" | grep -Eq '^gpu_temperature_celsius\{' || { echo "fabricated temp line: $output"; return 1; }
+}
+
+@test "nvidia csv yields util/vram/temp/power tagged vendor=nvidia" {
+    nvidia="$BATS_TEST_TMPDIR/nvidia-smi"
+    cat > "$nvidia" <<'STUB'
+echo '0, NVIDIA GeForce RTX 3090, 93, 23004, 24576, 76, 397.40'
+STUB
+    chmod +x "$nvidia"
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd "$nvidia"
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="nvidia"[^}]*\} 93$' || { echo "$output"; return 1; }
+    # memory.used MiB -> bytes (23004 * 1048576)
+    echo "$output" | grep -Eq 'gpu_memory_used_bytes\{[^}]*vendor="nvidia"[^}]*\} 24121442304$' || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_temperature_celsius\{[^}]*vendor="nvidia"[^}]*\} 76$'
+    echo "$output" | grep -Eq 'gpu_power_watts\{[^}]*vendor="nvidia"[^}]*\} 397\.4$'
+    echo "$output" | grep -Eq 'model="NVIDIA GeForce RTX 3090"'
+}
+
+@test "a failing nvidia command is tolerated: amd still emits and exit is 0" {
+    mk_amd 1 50 1000000 2000000 "" 40000
+    failing="$BATS_TEST_TMPDIR/nvidia-fail"
+    printf '#!/bin/sh\nexit 1\n' > "$failing"; chmod +x "$failing"
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd "$failing"
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    echo "$output" | grep -Eq 'gpu_utilization_percent\{[^}]*vendor="amd"[^}]*\} 50$'
+    ! echo "$output" | grep -q 'vendor="nvidia"'
+}
+
+@test "no GPUs at all yields only HELP/TYPE headers and exits 0" {
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ] || { echo "$output"; return 1; }
+    # families are declared even when empty, so the dashboard/queries stay valid.
+    echo "$output" | grep -q '# TYPE gpu_utilization_percent gauge'
+    echo "$output" | grep -q '# TYPE gpu_memory_used_bytes gauge'
+    ! echo "$output" | grep -Eq '^gpu_utilization_percent\{'
+}
+
+@test "output is valid Prometheus text" {
+    mk_amd 1 42 1000 2000 "" 50000
+    run python3 "$EXPORTER" --amdgpu-sysfs "$SYS" --nvidia-cmd false
+    [ "$status" -eq 0 ]
+    # every non-comment line is "<name>{labels} value" or "<name> value"
+    printf '%s\n' "$output" | python3 -c '
+import sys, re
+pat = re.compile(r"^[a-z_]+(\{[^}]*\})? [-+0-9.eE+]+$")
+for i, l in enumerate(sys.stdin):
+    l = l.rstrip("\n")
+    if not l or l.startswith("#"):
+        continue
+    assert pat.match(l), "malformed line %d: %r" % (i, l)
+print("prometheus text ok")
+'
+}
+
+# ---- deployment + wiring contracts -------------------------------------
+# The exporter is a HOST-PROCESS user unit (not a quadlet): it must read the
+# host's own DRM sysfs and shell to nvidia-smi, which a container could not do
+# without GPU passthrough. These tests pin the cross-file wiring a reader
+# cannot eyeball — a lone change on one side leaves the metric silently absent.
+
+@test "the base unit runs the exporter from the checkout and never binds a wildcard" {
+    local u="$REPO_ROOT/scripts/gpu-exporter.service"
+    [ -f "$u" ]
+    grep -q '^ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve$' "$u" \
+        || { echo "unit does not run the exporter --serve from the checkout"; return 1; }
+    grep -q '^Restart=always$' "$u"
+    grep -q '^WantedBy=default.target$' "$u"
+    # Base unit binds loopback only; the per-host LAN address is added by a
+    # deploy drop-in, so a checked-in unit is never a wildcard.
+    grep -q '^Environment=GPU_EXPORTER_BINDS=127.0.0.1$' "$u"
+    ! grep -qE '0\.0\.0\.0' "$u"
+}
+
+@test "the deployer installs, drops in the LAN bind, and health-gates gpu-exporter" {
+    local handler="$REPO_ROOT/scripts/deploy-remote.sh"
+    grep -q '^deploy_gpu_exporter() {' "$handler"
+    # Per-host LAN from $TARGET (never a hardcoded host, never a wildcard).
+    grep -q 'dankclaw) lan=192.168.0.143' "$handler"
+    grep -q 'dankserver) lan=192.168.0.235' "$handler"
+    # Drop-in binds loopback + this host's LAN as one comma list.
+    grep -q "GPU_EXPORTER_BINDS=127.0.0.1,%s" "$handler"
+    grep -q 'systemctl --user enable gpu-exporter.service' "$handler"
+    grep -q 'systemctl --user restart gpu-exporter.service' "$handler"
+    # Health gate waits on the loopback metrics endpoint.
+    grep -q 'http://127.0.0.1:9401/metrics' "$handler"
+    # Dispatched from the central case.
+    grep -q 'gpu-exporter) deploy_gpu_exporter ;;' "$handler"
+    # Allowlisted on BOTH hosts.
+    grep -q 'dankclaw:gpu-exporter' "$handler"
+    grep -q 'dankserver:gpu-exporter' "$handler"
+}
+
+@test "the planner selects gpu-exporter for its files on each target" {
+    local t f
+    for t in dankclaw dankserver; do
+        for f in scripts/gpu-exporter.py scripts/gpu-exporter.service; do
+            run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target "$t" --paths "$f"
+            [ "$status" -eq 0 ]
+            [ "$output" = "gpu-exporter" ] || { echo "$t/$f -> '$output'"; return 1; }
+        done
+    done
+}
+
+@test "Prometheus scrapes gpu-exporter where node/cadvisor are reachable on each host" {
+    # Cross-file contract, same posture as node-exporter: the local instance is
+    # reached via the podman gateway (this Prometheus is a bridge container and
+    # a host-process bind is refused at the host LAN IP); the Dankserver
+    # instance is a different machine, so plain LAN routing reaches it. If either
+    # address drifts from the exporter's bind, GPU panels go blank with no error.
+    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
+    grep -q 'job_name: "gpu-dankclaw"' "$cfg" || { echo "no gpu-dankclaw job"; return 1; }
+    grep -q 'job_name: "gpu-dankserver"' "$cfg" || { echo "no gpu-dankserver job"; return 1; }
+    grep -q '"host.containers.internal:9401"' "$cfg" \
+        || { echo "gpu-dankclaw does not scrape the podman gateway"; return 1; }
+    grep -q '"192.168.0.235:9401"' "$cfg" \
+        || { echo "gpu-dankserver does not scrape the LAN address"; return 1; }
+    # A vhost target would be red forever (Prometheus cannot solve Anubis).
+    ! grep -qE '"[^"]*franksdank[^"]*:9401"' "$cfg"
+}
+
+@test "the host-system dashboard graphs the gpu_* families the exporter emits" {
+    local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/host-system.json"
+    [ -f "$dash" ]
+    python3 -c "
+import json
+d = json.load(open('$dash'))
+exprs = [t['expr'] for p in d['panels'] for t in p.get('targets', [])]
+for want in ('gpu_utilization_percent', 'gpu_memory_used_bytes',
+             'gpu_temperature_celsius', 'gpu_power_watts'):
+    assert any(want in e for e in exprs), ('missing', want, exprs)
+print('gpu dashboard panels ok')
+"
+}
+
+@test "--serve answers on EVERY declared bind, not just the first" {
+    # The deployed unit serves loopback + the host LAN address, and Prometheus
+    # reaches the host through the gateway — which dials the LAN-class bind.
+    # A server that only runs serve_forever() on the first socket leaves the
+    # second one listen()ing-but-never-accepting: the deploy health gate on
+    # loopback passes, every scrape on the other bind dies on the SYN backlog,
+    # and no gate but this one sees it. (This exact bug shipped 2026-10-21 and
+    # left both gpu-* targets down beside a green deploy.)
+    mk_amd 0 42 1073741824 34359738368 "" 71000
+    port=$((19400 + RANDOM % 500))
+    python3 "$EXPORTER" --serve --amdgpu-sysfs "$SYS" --nvidia-cmd false \
+        --bind 127.0.0.1 --bind 127.0.0.2 --port "$port" \
+        > "$BATS_TEST_TMPDIR/serve.log" 2>&1 &
+    local pid=$!
+    trap 'kill "$pid" 2>/dev/null' EXIT
+    local ok=0
+    for _ in $(seq 1 20); do
+        curl -fsS -m2 "http://127.0.0.1:$port/metrics" >/dev/null 2>&1 && { ok=1; break; }
+        sleep 0.25
+    done
+    [ "$ok" -eq 1 ] || { kill "$pid" 2>/dev/null; return 1; }
+    # Both binds must ANSWER within 2 s each — the second one is the bug.
+    for b in 127.0.0.1 127.0.0.2; do
+        curl -fsS -m2 "http://$b:$port/metrics" | grep -q '^gpu_utilization_percent{' \
+            || { echo "bind $b did not answer /metrics"; kill "$pid" 2>/dev/null; return 1; }
+    done
+    kill "$pid" 2>/dev/null
+    trap - EXIT
+}
@@ -21,7 +21,7 @@ load helpers

 UNIT_DIR="$REPO_ROOT/dankclaw/observability/quadlet"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
-UNITS=(prometheus grafana loki alloy node-exporter alertmanager)
+UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager blackbox-exporter)

 # container_section <unit> — the [Container] key/value block only
 container_section() {
@@ -33,7 +33,7 @@ service_section() {
    awk '/^\[Service\]/{f=1;next} /^\[/{f=0} f' "$1"
 }

-@test "all six observability units exist under dankclaw/observability/quadlet/" {
+@test "all eight observability units exist under dankclaw/observability/quadlet/" {
    local svc
    for svc in "${UNITS[@]}"; do
        [ -f "$UNIT_DIR/$svc.container" ] || { echo "missing $svc.container"; return 1; }
@@ -277,6 +277,7 @@ loki http://127.0.0.1:3100/ready
 alloy http://127.0.0.1:12345/-/healthy
 node-exporter http://127.0.0.1:9100/metrics
 alertmanager http://127.0.0.1:9093/-/healthy
+cadvisor http://127.0.0.1:8080/healthz
 PAIRS
 }

@@ -568,3 +569,188 @@ assert any('ALERTS' in e for e in exprs), exprs
    [ "$status" -eq 0 ]
    [ "$output" = "alertmanager" ]
 }
+
+@test "cadvisor exports container metrics only because of --cgroupns=host, and binds exactly its two consumers" {
+    # Podman starts every container in a private cgroup namespace, so a cadvisor
+    # without --cgroupns=host sees only its own cgroup root and exports a single
+    # series (measured 2026-10-20: 1 series before the flag, 996 after). That
+    # flag and the host /sys mount are the entire mechanism — pin them, plus the
+    # publish set: loopback (host probe + health gate) and this host's LAN
+    # address (Prometheus reaches a host-published port there). Never a wildcard.
+    local unit lan
+    for pair in "dankclaw/observability/quadlet:192.168.0.143" "dankserver/observability/quadlet:192.168.0.235"; do
+        unit="$REPO_ROOT/${pair%%:*}/cadvisor.container"
+        lan="${pair##*:}"
+        [ -f "$unit" ] || { echo "missing $unit"; return 1; }
+        grep -q -- '--cgroupns=host' "$unit" || { echo "$unit lacks --cgroupns=host"; return 1; }
+        grep -q -- '--pid=host' "$unit" || { echo "$unit lacks --pid=host"; return 1; }
+        grep -q '^Volume=/sys:/sys:ro$' "$unit" || { echo "$unit lacks the host /sys mount"; return 1; }
+        grep -q '^WantedBy=default.target$' "$unit" || { echo "$unit is not boot-persistent"; return 1; }
+        grep -q '^Restart=always$' "$unit" || { echo "$unit is not self-healing"; return 1; }
+        grep -q '^HealthCmd=' "$unit" || { echo "$unit has no health check"; return 1; }
+        grep -q "^PublishPort=127.0.0.1:8080:8080\$" "$unit" || { echo "$unit must publish loopback :8080"; return 1; }
+        grep -q "^PublishPort=$lan:8080:8080\$" "$unit" || { echo "$unit must publish $lan:8080"; return 1; }
+        ! grep -qE '^PublishPort=(0\.0\.0\.0|:)' "$unit" || { echo "$unit publishes on a wildcard"; return 1; }
+    done
+}
+
+@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
+    # Cross-file contract, measured 2026-10-21 from inside the Prometheus
+    # container: a bridge container cannot dial its OWN host's LAN IP — the
+    # hairpin refuses published ports exactly like host-net binds, so
+    # cadvisor-dankclaw must scrape via the podman gateway (the original pin
+    # here demanded the LAN address and shipped a permanently-down target).
+    # dankserver is a different machine: plain LAN there. And the whole
+    # dashboard depends on the id -> container relabel: cadvisor names nothing
+    # itself, the quadlet unit name is only in the cgroup path. If either half
+    # changes alone, the Containers dashboard is blank.
+    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
+    grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
+    grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
+    grep -q '"host.containers.internal:8080"' "$cfg" \
+        || { echo "cadvisor-dankclaw does not scrape via the podman gateway"; return 1; }
+    ! grep -q '"192.168.0.143:8080"' "$cfg" \
+        || { echo "cadvisor-dankclaw dials its own host LAN IP — hairpin refuses it"; return 1; }
+    grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
+    # The relabel that turns …/app.slice/<name>.service… into a `container`.
+    grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }
+    grep -q 'target_label: container' "$cfg" || { echo "relabel does not set the container label"; return 1; }
+}
+
+@test "the Containers dashboard is provisioned from git" {
+    local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/containers.json"
+    [ -f "$dash" ]
+    python3 -c "
+import json
+d = json.load(open('$dash'))
+assert d['uid'] == 'containers', d['uid']
+names = [t['expr'] for p in d['panels'] for t in p.get('targets', [])]
+# Only families cadvisor actually exports for rootless quadlet containers are
+# graphed; network/fs are absent for them, so a network/fs panel would be a
+# permanent empty lie.
+assert any('container_cpu_usage_seconds_total' in e for e in names), names
+assert any('container_memory_working_set_bytes' in e for e in names), names
+assert not any('container_network' in e or 'container_fs_' in e for e in names), names
+"
+}
+
+@test "the planner selects cadvisor for its unit on each target" {
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
+        --paths dankclaw/observability/quadlet/cadvisor.container
+    [ "$status" -eq 0 ]
+    [ "$output" = "cadvisor" ]
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver \
+        --paths dankserver/observability/quadlet/cadvisor.container
+    [ "$status" -eq 0 ]
+    [ "$output" = "cadvisor" ]
+}
+
+@test "the grafana copier provisions EVERY dashboard json, not a hand-listed subset" {
+    # Behavioral, not a text grep: the copier is extracted and executed with a
+    # stubbed run_privileged against a sandbox. The fleet provider turns each
+    # file in its directory into a dashboard, so a file the copier skips is a
+    # dashboard that silently never exists (the containers board lived in the
+    # repo, never reached /opt, and its fixes were undeployable).
+    # BATS_TEST_TMPDIR: per-test, removed by bats itself; a mktemp dir with a
+    # RETURN trap races bats' own trap machinery and vanishes mid-function.
+    local sandbox="$BATS_TEST_TMPDIR/sandbox"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_grafana_provisioning()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        export GRAFANA_PROV_ROOT="$sandbox/opt/grafana"
+        copy_grafana_provisioning
+    )
+    local f
+    for f in "$REPO_ROOT"/shared/grafana/provisioning/dashboards/*.json; do
+        [ -f "$sandbox/opt/grafana/provisioning/dashboards/$(basename "$f")" ]
+    done
+    [ -f "$sandbox/opt/grafana/provisioning/dashboards/provider.yml" ]
+    [ -f "$sandbox/opt/grafana/provisioning/datasources/prometheus.yml" ]
+}
+
+@test "changes under shared/blackbox/ plan the prober" {
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
+        --paths shared/blackbox/blackbox.yml
+    [ "$status" -eq 0 ]
+    [ "$output" = "blackbox-exporter" ]
+}
+
+@test "the copier runs the same code path the deployer runs" {
+    # Textually pinned so a refactor cannot quietly route the deployer around
+    # the copy_* functions the seams execute.
+    grep -q '        prometheus)' "$REMOTE_DEPLOY"
+    grep -q 'copy_prometheus_config' "$REMOTE_DEPLOY"
+    grep -q 'copy_alertmanager_config' "$REMOTE_DEPLOY"
+    grep -q 'copy_loki_config' "$REMOTE_DEPLOY"
+    grep -q '        blackbox-exporter)' "$REMOTE_DEPLOY"
+    grep -q 'copy_blackbox_config' "$REMOTE_DEPLOY"
+}
+
+@test "the alloy copier installs the host-labeled config, token-free" {
+    # Executes copy_alloy_config for real against a sandbox (stubbed
+    # run_privileged). A surviving GRAFANA_HOST_LABEL token would ship a
+    # config whose host label is a literal placeholder — the exact class of
+    # silent-label defect this tranche retired the board for.
+    local sandbox="$BATS_TEST_TMPDIR/opt"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        # NOTE: VAR=x func does not bind for shell functions — export.
+        export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
+        copy_alloy_config
+    )
+    [ -f "$sandbox/config/config.alloy" ]
+    grep -q 'host = "dankclaw",' "$sandbox/config/config.alloy"
+    ! grep -q "GRAFANA_HOST_LABEL" "$sandbox/config/config.alloy"
+}
+
+@test "the blackbox copier installs the prober config" {
+    local sandbox="$BATS_TEST_TMPDIR/blackbox"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_blackbox_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        export BLACKBOX_CONF_ROOT="$sandbox"
+        copy_blackbox_config
+    )
+    [ -f "$sandbox/blackbox.yml" ]
+    grep -q "listen_address: 127.0.0.1:9115" "$sandbox/blackbox.yml"
+}
+
+@test "the alloy copier refuses to install a config the engine rejects" {
+    # The syntax gate: alloy attributes are position-sensitive and a
+    # misplaced one only explodes at run time (crash-loop behind a green
+    # deploy). Prove the copier refuses a config the pinned image won't
+    # validate, and installs nothing. Needs the alloy image; skipped
+    # where it is absent (the copier's gate is likewise image-gated).
+    if ! podman image exists localhost/franksdank/alloy:latest; then
+        skip "franksdank/alloy image not present on this box"
+    fi
+    local sandbox="$BATS_TEST_TMPDIR/opt" cfg="$BATS_TEST_TMPDIR/broken.alloy"
+    sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
+    printf 'bogus_attr = "x"\n' >> "$cfg"
+    ! (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        cp() { if [ "$1" = "$cfg" ]; then return 1; fi; command cp "$@"; }
+        export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
+        copy_alloy_config
+    )
+    [ ! -f "$sandbox/config/config.alloy" ]
+}
+
+@test "the installed alloy config is engine-valid" {
+    # Runs the pinned image's own validator over the shipped config. The
+    # 2026-10-21 defect (`on_error` on the labels stage, which alloy 1.21
+    # does not define) passed every textual seam and only failed at boot.
+    if ! podman image exists localhost/franksdank/alloy:latest; then
+        skip "franksdank/alloy image not present on this box"
+    fi
+    local cfg="$BATS_TEST_TMPDIR/rendered.alloy"
+    sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
+    podman run --rm -v "$cfg:/c.alloy:ro" --entrypoint /usr/bin/alloy \
+        localhost/franksdank/alloy:latest validate /c.alloy
+}
Author	SHA1	Message	Date
Frank	e6fcf217b4	docs(comfyui): the GTT aperture is not free RAM beside frank-dragoon Measured 2026-10-21 with frank-dragoon live: MemAvailable 15.3 GiB, iGPU busy 1.75 GiB. The Qwen-Image weight sets (13.3-14.2 GB) do not fit that budget, and the page's 122 GB /system_stats figure is the aperture, not headroom. Adds the fit table and the stop-mask-render-verify procedure (the 3090 bench law, applied to the APU); ComfyUI.md gotcha links it.	2026-10-21 13:35:50 -07:00
Frank	cb86ad8fd7	docs(observability): post-blackbox state, copier law, label-promotion why Prometheus/Observability/Provisioning pages: every job up after the blackbox swap, the copier list learns copy_blackbox_config and the alloy validate gate, the config table gains alloy + blackbox rows, and the logs bullet states the stream-label law instead of the retired 'awaiting deploy' note.	2026-10-21 13:33:03 -07:00
Frank	078b550bd8	test(observability): copier sandboxes, probe-target seam exception, cop gates - caddyfile seam learns the documented exception: URL-shaped targets are legal only inside a prober job (state is per-job; the first cut leaked it and passed a planted URL). - observability seams: blackbox unit + planner entries, both copiers executed for real in sandboxes, alloy engine-validation refusal (image-gated), dashboard panel invariants. - AGENTS.md: structural target downs retired.	2026-10-21 13:33:03 -07:00
Frank	765a8cb7de	feat(deploy): blackbox copier, alloy engine-validation gate, planner wiring copy_alloy_config now validates the rendered config with the very alloy image that will run it, before installing: a misplaced attribute crash-loops the unit behind a green deploy, and no textual seam can see it. copy_blackbox_config ships the prober config; the planner maps the new unit and shared/blackbox/** to the blackbox-exporter service, ordered before prometheus.	2026-10-21 13:33:03 -07:00
Frank	999c4fe440	fix(alloy): promote CONTAINER_NAME to an indexed Loki stream label Loki metrics queries group only by indexed stream labels, so the journal-as-JSON stream made every per-container view structurally empty (measured 6.5k lines/5min flowing while the board read zero). A loki.process relabel stage now extracts CONTAINER_NAME into a 'container' label. alloy 1.21's json stage has no on_error attribute (validated against the pinned image) — format_as_json guarantees the line shape, so no error branch exists.	2026-10-21 13:33:03 -07:00
Frank	2af2e6ac56	fix(grafana): observability and logs-overview boards read real data observability.json: stale 'structural downs' panel description fixed, target census matches the post-blackbox job set. logs-overview.json: panels grouped by the promoted container label instead of the absent container_name, so the board stops reading empty beside live traffic.	2026-10-21 13:32:58 -07:00
Frank	fbe54ca1ad	feat(gitea): enable /metrics for Prometheus scraping Gitea shipped metrics disabled, so its scrape target was a permanent structural down beside a healthy service.	2026-10-21 13:32:58 -07:00
Frank	f073b91033	feat(observability): blackbox exporter replaces the HTML-probe jobs Prometheus 3 refuses HTML scrape bodies, so the comfyui/services probe jobs were structurally dead targets reading as outages. HTTP liveness now rides a blackbox exporter (shared/blackbox/blackbox.yml, loopback bind, http_2xx module); prometheus.yml probes SearXNG, OpenWebUI and ComfyUI via __param_target relabeling. The unit is a boot-persistent, self-healing root quadlet.	2026-10-21 13:32:58 -07:00
dank	425994515a	Merge pull request 'fix(deploy): grafana copier provisions EVERY dashboard json' (#68 ) from fix/grafana-copier-glob into main	2026-10-21 09:49:15 -07:00
Frank	c41384e03d	fix(deploy): grafana copier provisions EVERY dashboard json, not a hand-listed subset copy_grafana_provisioning named llm-fleet.json alone, so a new board in shared/grafana/provisioning/dashboards/ silently never reached /opt/grafana/provisioning. Measured 2026-10-21: the containers board was merged, seam-tested, and documented as working since 2026-10-20, yet was NEVER provisioned — its panel fixes were undeployable; the other four boards existed on disk only because someone hand-copied them once (and host-system.json there had gone stale against the repo). The copier now globs the directory, and GRAFANA_PROV_ROOT lets the new bats case run the copier for real into a sandbox (red against the old hand-listed body, green against the glob). Vault: the copier law + the containers incident on Observability Provisioning, the corrected dashboards claim on Observability. Note: a mktemp sandbox + trap RETURN races bats' trap machinery (the dir vanishes mid-function); BATS_TEST_TMPDIR is the safe idiom.	2026-10-21 09:40:57 -07:00
dank	127621bdd7	Merge pull request 'docs(vault)+fix(grafana): audit tranche 1 and dashboard data-flow fixes' (#67 ) from fix/dragoon-context-200k into main	2026-10-21 09:28:43 -07:00
Frank	c96d0228a4	fix(grafana): dashboard census fixes — every panel now receives data Census: all 5 provisioned dashboards, 56 targets, every expression re-evaluated against the live Prometheus/Loki with template vars expanded (queries must run from the host: both APIs bind 127.0.0.1; from the dev box that is connection-refused, which is NOT a dashboard defect). - host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops non-matching pairs and the right side collapses to {job} while the left carries {job,instance} -> 0 series forever. Fixed with / on(job); proven live (2 series). - containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits), so the '> 0' filter can never match. Rewritten to share-of-host-total via label_replace join of the node exporter's MemTotal by host (the parens form is a PromQL parse error: nothing may follow group_left); 62 series live. Panel unit percentunit kept -> expression returns a fraction. - llm-fleet 'Firing fleet alerts' is legitimately empty when healthy (ALERTS carries only firing/pending series); description now says so, per the Stale Signal rule that 0-series-on-ALERTS is not a defect. logs-overview and observability: all targets already returning data.	2026-10-21 02:45:57 -07:00
Frank	a61308d72e	docs(vault): audit tranche 1 — one-owner dedupe of the observability cluster, dead-shape numbers destroyed Purpose-test pass over the audit findings (every verdict re-verified against the file before touching it; three scout findings were hallucinated — a nonexistent filename, my own changelog line, and sanctioned Eras rows): - Observability Provisioning: destroyed the 63-line 'Current delivered state' block — chain topology/relay/delivery restates [[Observability]] and [[Alertmanager]] (those are the owners). Its three sole-custody laws MOVED to Stale Signal Audit as 'Three rules that outlive the incident': down-beside-healthy -> read lastError; Prometheus alert state in process vs Alertmanager's --storage.path persistence; candidate-arm metrics are never fleet state. - RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool; never reuse a B/token ratio across geometries; pin pool bytes floor-sized), era-labelled the rates, destroyed the measured ladder table for the superseded shape-G/bf16 rungs. - Stale Signal Audit: new rules section + Provisioning Related link. check-okf 109/0, vault links 0 unresolved, docs seams green.	2026-10-21 02:35:44 -07:00
dank	ccb805cf0b	Merge pull request 'fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches' (#66 ) from fix/dragoon-context-200k into main	2026-10-21 02:25:42 -07:00
Frank	c6e2887f94	fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches The lane's child dlopens libxrt_core.so.2 from /opt/xilinx/lib/<multiarch>/ as well; our mounts filled only /opt/xilinx/xrt/lib and the install path. Since the host's libxrt2 was reconfigured 2026-10-20 the child dies naming the missing prefix ('No such library /opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2') and the engine supervisor takes the WHOLE container down on a dead lane — measured 2026-10-21: frank-dragoon boot-looped 88 times with chat dead while the GPU lines looked healthy. Each XRT library now mounts THREE times from the same real file (never a symlink); the multiarch name derives from FRANK_DRAGOON_XRT_DIR's basename so the bats fixture proves the derivation. Proven end-to-end before commit: production-shaped scratch boot + the three extra mounts answers the full boot ledger, npu_api buckets u0..u9, and /health ok at the 262144 ceiling. A dead lane is not survivable on this engine — arm it only with all three prefixes (page law updated).	2026-10-21 02:21:53 -07:00
dank	9eb53ef937	Merge pull request 'feat(frank-dragoon): 200,000-token harness window via the native 262,144 ceiling' (#65 ) from fix/dragoon-context-200k into main	2026-10-21 01:40:40 -07:00
Frank	688928d489	docs(dragoon): the 262144 ceiling, the 200,000 window, and the 95 % trigger law Shape + harness contract on the endpoint page, AGENTS.md, and the OMP Harness worked example recomputed to the live pair; the 400-zone law now names the 85 %-comments vs 95 %-config disagreement as the recurring cause and points at the seam that enforces the arithmetic.	2026-10-21 01:34:28 -07:00
Frank	56503dbad1	feat(frank-dragoon): raise HALOGEN_CTX to the native 262144 ceiling The 200,704 ceiling squeezed the harness contextWindow down to 155,648 (the 400-zone fit law needs ~45k of gap under the ceiling), which is why long dragoon sessions kept hitting the 400 zone: the harness clamps max_tokens to window - est - 4096, its chars/4 est under-counts ~3 %, and config.yml compacts at 95 % while the pair was designed for 85 %. Ceiling -> checkpoint's native 262,144 (no RAM move: the KV pool stays 401,408 and is what the ledger bills; the band ctx <= pool <= slots*ctx still holds). The harness window becomes 200,000 with a 62,144 gap. The fit-law seam now reads the declared window from the vault page (the one owner) and checks the arithmetic at the live 95 % trigger, so a ceiling/window pair that drifts back together fails tests/run.sh instead of a live session.	2026-10-21 01:34:28 -07:00
dank	fbeb9701e4	Merge pull request 'fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway' (#64 ) from fix/gpu-serve-and-cadvisor-target into main	2026-10-21 00:39:54 -07:00
Frank	650b6e0281	fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and cadvisor-dankclaw DOWN beside a green deploy: - gpu-exporter.py built one ThreadingHTTPServer per bind but ran serve_forever() on servers[0] only: the LAN-class socket sat listen()ing but never accept()ed, loopback deploy gate passed, every scrape that reached the other bind timed out on the SYN backlog. Now one thread per extra bind. Regression test spins --serve on two loopback binds and requires BOTH to answer /metrics. - cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the Prometheus container: a bridge container cannot dial its own host's LAN address at all (hairpin refuses published ports too); the podman gateway answers. Target moved to host.containers.internal:8080; the seam that pinned the LAN address (asserting the wrong doctrine) is re-pinned to the gateway plus a negative pin against the own-host LAN dial. Observability page records the measured reachability law replacing the inferred one.	2026-10-21 00:34:17 -07:00
dank	b62a05837f	Merge pull request 'fix(deploy): client allowlist missing cadvisor/gpu-exporter/prowlarr' (#63 ) from fix/deploy-sh-allowlist-drift into main	2026-10-21 00:15:45 -07:00
Frank	0367244d2a	fix(deploy): client allowlist was missing cadvisor, gpu-exporter, prowlarr deploy.sh's valid_service and deploy-remote.sh's service_is_allowed are two gates for the same door, and they drifted: the remote gate gained cadvisor (2026-10-20) and gpu-exporter (2026-10-21) while the client gate never did, so `--service cadvisor\|gpu-exporter` died with "is not deployed on <target>" at the client — the first-ever deploy of a new service is exactly the case that needs the client gate to know it. dankserver:prowlarr (planner can emit it for dankserver) was remote-only too. Adds the four missing pairs to the client gate and a deployment_flow contract test that diffs the two extracted lists per target — the test the cadvisor PR should have carried. searxng-network stays the one sanctioned client-only exception (network unit, no remote handler), documented inline.	2026-10-21 00:15:21 -07:00
dank	656d9567d5	Merge pull request 'chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest)' (#62 ) from chore/plex-1435 into main	2026-10-21 00:07:28 -07:00
Frank	0b4bc60a38	chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest) ls329 is the newest build on the linuxserver channel and Plex 1.43.5.11029 is their newest release; the live pin (ls325 / 1.43.4.10903) is two Plex releases behind. Digest-verified at pin time: the tag's manifest digest equals the registry's current 'latest' (sha256:8f655bad...e9e2334), the same law the page records. Quadlet restart runs PMS's usual one-minor schema migration on first boot; the deployer's /identity HTTPS gate tolerates the first-boot migration up to 600 s. Rollback: revert this commit (one line).	2026-10-21 00:06:53 -07:00
dank	a2011f4eb6	Merge pull request 'feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet' (#61 ) from feat/gpu-stats into main	2026-10-20 23:50:11 -07:00
Frank	b02647b992	feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet Add a host-process GPU exporter, scrape it on both servers, and graph it on the host-system dashboard. The fleet has four GPUs across three vendors of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw) and nothing exposed them until now. scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs (gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average) plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus families (utilization_percent, memory_used/total_bytes, temperature_celsius, power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series, never a fabricated 0: the R9700s expose no sysfs power, so they simply carry no gpu_power_watts line. Both sources are injectable, so the seam runs on a GPU-less dev box; live-verified on Dankserver's real sysfs. It runs as a plain user unit (not a quadlet): it must read the host's own DRM sysfs and shell to nvidia-smi, which a container cannot do without GPU passthrough. Same posture as signal-relay. The base unit binds loopback; the deploy drop-in appends the host LAN address, never a wildcard (matching node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it on both hosts and is wired into the planner (both targets), the test-changed router, and two Prometheus scrape jobs using the node/cadvisor reach rules (gateway on Dankclaw, LAN on Dankserver).	2026-10-20 23:42:26 -07:00
dank	b35269bfa5	Merge pull request 'docs(archon): four-slot ledger, resolver law from source, probe self-calibration' (#60 ) from wt/20261020-213526 into main	2026-10-20 23:30:27 -07:00
dank	7993a336bf	Merge pull request 'feat(frank-archon): four decode slots over a >=700k-token KV pool' (#59 ) from wt/20261020-213526 into main	2026-10-20 23:14:35 -07:00
dank	31b71a16d6	Merge pull request 'feat(observability): per-container metrics via cadvisor + Containers dashboard' (#58 ) from feat/containers-observability into main	2026-10-20 23:08:23 -07:00
dank	a4edaa7d9d	Merge pull request 'feat(frank-archon): arm --prefill-interleave at 1' (#57 ) from wt/20261020-213526 into main	2026-10-20 22:55:58 -07:00
Frank	31908c623f	feat(observability): per-container metrics via cadvisor + Containers dashboard Per-container CPU/memory was invisible for the quadlet app stack — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a `containers` dashboard. cadvisor works on this rootless-Podman / cgroup-v2 host on one condition: --cgroupns=host. Podman starts every container in a private cgroup namespace, so without it cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the rootless app containers (it resolved 2 of 34), so naming comes from elsewhere: the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into a `container` label. No podman socket needed. cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 and the host LAN address — never a wildcard. Prometheus (a bridge container) reaches a host-published port at the LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports for these containers are graphed — CPU usage, memory working-set/limit, OOM — not network or filesystem, which are absent under the host-PID layout. Wired through the deployer (allowlist, deploy_observability case + /healthz gate, dispatcher), the planner (both targets), and observability_quadlet.bats.	2026-10-20 22:51:42 -07:00