Merge pull request 'fix(deploy): relay firewall DROP must be appended, not -I-inserted above its ACCEPTs' (#75 ) from fix/signal-relay-fw-order into main

fix(deploy): relay firewall DROP must be appended, not -I-inserted above its ACCEPTs
iptables -I prepends, so the install block's final -I of the blanket DROP landed it at position 1, above all three ACCEPTs: every packet to 18093/tcp died there (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT). The host's own health probe timed out beside a LISTENing socket and the whole dankclaw drain aborted at the alertmanager gate twice. Install now rebuilds the tail: delete any blanket DROP wherever it sits, keep the ACCEPTs at the head, append the DROP last - converges on hosts poisoned by the old order. New bats case runs install_signal_relay for real against an iptables sandbox seeded with the poisoned order (red on the old code, green on this one).
2026-10-21 16:59:16 -07:00 · 2026-10-21 16:52:40 -07:00 · 2026-10-21 16:48:32 -07:00 · 2026-10-21 16:32:44 -07:00 · 2026-10-21 16:32:36 -07:00 · 2026-10-21 16:32:36 -07:00
61 changed files with 1265 additions and 252 deletions
@@ -176,11 +176,12 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
 (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
 IP, the one bind a container can reach as `host.containers.internal`;
 iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
-Prometheus targets read down for structural reasons (HTML-scraped "probes",
-Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
-a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
-without `--metrics`. See
-`serverAppConfigs-docs/Infrastructure/Observability.md`.
+The former structural target downs (HTML-scraped "probes", Gitea metrics off)
+are retired as of 2026-10-21: HTTP liveness rides a blackbox exporter
+(`shared/blackbox/blackbox.yml`), Gitea serves `/metrics` from its quadlet, and
+every Prometheus job reports up. `frank-zealot` was down for a different reason
+until 2026-10-18: llama.cpp will not serve `/metrics` at all without
+`--metrics`. See `serverAppConfigs-docs/Infrastructure/Observability.md`.

 ## Testing

@@ -20,7 +20,7 @@ Wants=authentik-server.service

 [Container]
 ContainerName=authentik-outpost
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -14,7 +14,7 @@ After=network.target

 [Container]
 ContainerName=authentik-postgres
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
 Environment=POSTGRES_DB=authentik
@@ -19,7 +19,7 @@ Wants=authentik-postgres.service

 [Container]
 ContainerName=authentik-server
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
 Exec=server
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -16,7 +16,7 @@ Wants=authentik-server.service

 [Container]
 ContainerName=authentik-worker
-Image=ghcr.io/goauthentik/server:2026.8.3
+Image=ghcr.io/goauthentik/server:2026.8.3@sha256:ab9b4e8cc4ab3f8d1198d2db6aeea66bafea1963b3f2843589e0d163f97d9849
 Exec=worker
 Network=dankclaw_default
 EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
@@ -15,7 +15,7 @@ After=network.target
 [Container]
 ContainerName=gitea-postgres
 Network=gitea.network
-Image=docker.io/library/postgres:16-alpine
+Image=docker.io/library/postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=POSTGRES_USER=gitea
 Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}
@@ -19,7 +19,7 @@ After=gitea-postgres.service network.target
 [Container]
 ContainerName=gitea-server
 Network=gitea.network
-Image=docker.io/gitea/gitea:latest
+Image=docker.io/gitea/gitea:latest@sha256:30777c91280d973fa02be48b1ade735bd6859e8f6f7af3b4444d519437852504
 EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=DATABASE_TYPE=postgres
 Environment=DATABASE_HOST=gitea-postgres
@@ -29,6 +29,13 @@ Environment=DATABASE_PASSWD=${GITEA_DB_PASSWORD}
 Environment=DATABASE_PORT=5432
 Environment=USER_UID=1000
 Environment=USER_GID=1000
+# Prometheus scrapes /metrics over the published LAN port (job "gitea" in
+# shared/prometheus/prometheus.yml). Off, /metrics is a 404 and the target is
+# permanently red (the audit's "disabled metrics" class). 0.0.0.0 binds inside
+# the container; the only route in is this host's published 3000, and the
+# exposed surface is counters only — same posture as the LLM /metrics.
+Environment=GITEA__metrics__ENABLED=true
+Environment=GITEA__metrics__ENABLED_HOST=0.0.0.0
 PublishPort=0.0.0.0:3000:3000
 PublishPort=2222:22
 Volume=/home/dank/Containers/gitea/data:/data
@@ -23,7 +23,7 @@ After=gluetun.service network.target
 [Container]
 ContainerName=marreta_browser
 Network=gluetun.container
-Image=docker.io/lightpanda/browser:nightly
+Image=docker.io/lightpanda/browser:nightly@sha256:979c36ed869f1016b8dcf22939a0633c262f2c30e892a03a9f720eba2ff1931e

 [Service]
 TimeoutStartSec=300
@@ -32,7 +32,7 @@ After=gluetun.service marreta-browser.service network.target
 [Container]
 ContainerName=marreta
 Network=gluetun.container
-Image=ghcr.io/manualdousuario/marreta:latest
+Image=ghcr.io/manualdousuario/marreta:latest@sha256:30f2a34567b79b66da794b4b9423751bbbcd07ceff600d9f6a2d219e7e439362
 # Relocate off SearXNG's 8080 inside the shared netns.
 Environment=NGINX_HTTP_PORT=8085
 Environment=APP_NAME=Marreta
@@ -33,7 +33,7 @@ Wants=network-online.target

 [Container]
 ContainerName=alertmanager
-Image=docker.io/prom/alertmanager:latest
+Image=docker.io/prom/alertmanager:latest@sha256:e9733bafb1bdef9b00e25a21f8f99dc26a22224bf16641ad754d1649f4c3357a
 Network=dankclaw_default
 User=0:0
 Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager
@@ -0,0 +1,56 @@
+# Quadlet unit: blackbox exporter (rootless, host network, boot-persistent).
+#
+# Install path on dankclaw:
+#   ~/.config/containers/systemd/blackbox-exporter.container
+#
+# WHY Network=host. The probes it runs are (a) the host's own LAN IP
+# (192.168.0.143:8081 SearXNG, :3005 OpenWebUI — host-process/portmap binds a
+# bridge container cannot reach; netavark DNATs published ports only, see
+# Container Host Reachability in the vault) and (b) ComfyUI's published port
+# via 127.0.0.1 (rootless --network=host shares the real host netns, so the
+# 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
+# reach neither class reliably.
+#
+# WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial
+# its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a
+# bridge container dialing the podman gateway 169.254.1.2 lands on the host
+# LOOPBACK — that is the scrape path, and it needs this exporter to be
+# LISTENING ON LOOPBACK (proved: 200 from the prometheus container to
+# http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway
+# address itself is NOT bindable from the host netns — it exists only
+# inside each rootless netns; binding it dies "bind: cannot assign
+# requested address" (live 2026-10-21). The LAN bind stays for diagnostics.
+# No public DNS name, no guard chain: internal net only, per the exposure
+# policy. The listen flag is --web.listen-address and it is REPEATABLE, one
+# flag per bind; plain --listen-address does not exist and the exporter
+# exits in 100 ms with "unknown long flag" (live 2026-10-21).
+
+[Unit]
+Description=Blackbox exporter (HTTP probe prober)
+After=network-online.target
+
+[Container]
+Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
+ContainerName=blackbox-exporter
+Network=host
+# The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live
+# 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries
+# FLAGS ONLY — repeating the binary makes it a positional arg and kingpin
+# exits "unexpected /bin/blackbox_exporter".
+Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115
+Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+# Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`)
+# is passed as one quoted token and quadlet then folds the following
+# --health-interval/--health-retries (and the image name) into the command
+# string — unit died twice that way, live 2026-10-21. wget only; this
+# busybox variant has no curl, and host-network wget answers on loopback.
+HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy
+HealthInterval=30s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target
@@ -35,7 +35,7 @@ Wants=network-online.target

 [Container]
 ContainerName=cadvisor
-Image=gcr.io/cadvisor/cadvisor:v0.49.1
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
 PodmanArgs=--pid=host --cgroupns=host
 PublishPort=127.0.0.1:8080:8080
 PublishPort=192.168.0.143:8080:8080
@@ -34,7 +34,7 @@ Wants=network-online.target prometheus.service

 [Container]
 ContainerName=grafana
-Image=docker.io/grafana/grafana:latest
+Image=docker.io/grafana/grafana:latest@sha256:dbbc38ea21103cefb50396340cd88afec779c7e070b0a7bf5a75b4ce46651d3f
 Network=dankclaw_default
 User=0:0
 PublishPort=127.0.0.1:3002:3000
@@ -33,7 +33,7 @@ Wants=network-online.target

 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100
@@ -43,8 +43,16 @@ Wants=network-online.target

 [Container]
 ContainerName=prometheus
-Image=docker.io/prom/prometheus:latest
+Image=docker.io/prom/prometheus:latest@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
 Network=dankclaw_default
+# Same-host scrape targets are dialed at host.containers.internal (the podman
+# gateway): a bridge container cannot dial its own host's LAN IP (netavark
+# hairpin gap, measured 2026-10-21), and traffic to the gateway lands on host
+# loopback / published ports (cadvisor/gpu jobs and the blackbox scrape all
+# ride it). The name resolves natively — NO HostAddr key: quadlet 5.7.0 calls
+# it unsupported and the WHOLE user quadlet batch then fails to generate,
+# silently orphaning every other unit (live 2026-10-21: prometheus.service
+# vanished while its container kept running as an orphan).
 User=0:0
 Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
 PublishPort=127.0.0.1:9090:9090
@@ -27,7 +27,7 @@ After=network.target
 [Container]
 ContainerName=open-webui
 Network=open-webui.network
-Image=ghcr.io/open-webui/open-webui:main
+Image=ghcr.io/open-webui/open-webui:main@sha256:4860ba005ae8f19f08d1e12b0868855b7b339ea0dad9927f196820fc0de04575
 PublishPort=3005:8080
 Environment=OPENAI_API_BASE_URL=http://192.168.0.143:7770
 Environment=ANONYMIZED_TELEMETRY=False
@@ -24,7 +24,7 @@ Wants=network-online.target

 [Container]
 ContainerName=pihole
-Image=docker.io/pihole/pihole:latest
+Image=docker.io/pihole/pihole:latest@sha256:d34134716fc32038b052a8674c7c6879e0ff21f0e6d8af345166fffa2c27c113
 Network=slirp4netns
 PublishPort=0.0.0.0:53:53/udp
 PublishPort=0.0.0.0:53:53/tcp
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 [Container]
 ContainerName=gluetun
 Network=searxng.network
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
@@ -16,7 +16,7 @@ After=searxng-network.service network.target
 ContainerName=searxng-valkey
 Network=searxng.network
 NetworkAlias=valkey
-Image=docker.io/valkey/valkey:9-alpine
+Image=docker.io/valkey/valkey:9-alpine@sha256:48332870af354a799964c0012ae1194a0bf2bf894eb508f945810596dc2d8d11
 Volume=/opt/searxng/valkey:/data

 [Service]
@@ -16,7 +16,7 @@ After=gluetun.service searxng-valkey.service network.target
 [Container]
 ContainerName=searxng
 Network=gluetun.container
-Image=ghcr.io/searxng/searxng:latest
+Image=ghcr.io/searxng/searxng:latest@sha256:5ed0323b5f5f8701264ef65e239a2bd83cb57098f2b971fa30bf79c06eeae271
 Environment=SEARXNG_SECRET=${SEARXNG_SECRET}
 Environment=SEARXNG_BASE_URL=${SEARXNG_BASE_URL}
 Volume=/opt/searxng/config:/etc/searxng
@@ -29,7 +29,7 @@ Wants=network-online.target

 [Container]
 ContainerName=seerr
-Image=ghcr.io/seerr-team/seerr:v3.4.1
+Image=ghcr.io/seerr-team/seerr:v3.4.1@sha256:f4768de5f616248d723e05891f3345a1402123775d03bf0890dbfedc0831bda1
 Network=dankclaw_default
 PublishPort=127.0.0.1:5055:5055
 Volume=/home/dank/Containers/seerr/data:/app/config
@@ -43,7 +43,7 @@ After=network.target

 [Container]
 ContainerName=calibre-server
-Image=lscr.io/linuxserver/calibre:9.15.0
+Image=lscr.io/linuxserver/calibre:9.15.0@sha256:59e4b0e036e62f228f8bcfae1ade42f47047719885fe454ddbad38e4597b987b
 Network=pasta
 EnvironmentFile=/home/dank/Containers/calibre-server/env
 Environment=PUID=1000
@@ -16,7 +16,7 @@ After=network.target

 [Container]
 ContainerName=calibre-web
-Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402
+Image=lscr.io/linuxserver/calibre-web:0.6.27-ls402@sha256:802729966faed5e32d671cd4f92ba6196e9e754a6c0f740ce1ee9acd86b10e55
 Network=pasta
 Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
 Environment=LANG=en_US.UTF-8
@@ -21,7 +21,13 @@ After=network.target

 [Container]
 ContainerName=copyparty
-Image=ghcr.io/9001/copyparty-ac:latest
+# Digest-pinned: upstream stopped publishing version tags after 1.19.15, so
+# the pin rides :latest@digest (the tag in the reference is cosmetic; the
+# digest is the pull). Pinned 2026-10-21 at the build whose label is
+# 1.20.26 - closes GHSA-3c88-p299-7hvr (1.20.25) and GHSA-hg99-jqfp-rf76
+# (1.20.26); the previously running build was the rolling :latest at
+# 1.20.24 (issue #73 on command-center).
+Image=ghcr.io/9001/copyparty-ac:latest@sha256:7f77459160a4bf34fdca79f2965dff4f0c1d7069a3d9cdfb3632acb041dd2f6a
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/copyparty/env
 Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank
@@ -16,7 +16,7 @@ After=network.target

 [Container]
 ContainerName=jellyfin
-Image=ghcr.io/jellyfin/jellyfin:latest
+Image=ghcr.io/jellyfin/jellyfin:latest@sha256:c3efa69a22f2691fc88d6bb035e87a8560e395a6047f062fb381c61c3185e8f5
 Network=dankserver_appnet
 Environment=DEBIAN_FRONTEND=noninteractive
 Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
@@ -18,7 +18,7 @@ Wants=network-online.target

 [Container]
 ContainerName=cadvisor
-Image=gcr.io/cadvisor/cadvisor:v0.49.1
+Image=gcr.io/cadvisor/cadvisor:v0.49.1@sha256:3cde6faf0791ebf7b41d6f8ae7145466fed712ea6f252c935294d2608b1af388
 PodmanArgs=--pid=host --cgroupns=host
 PublishPort=127.0.0.1:8080:8080
 PublishPort=192.168.0.235:8080:8080
@@ -37,7 +37,7 @@ Wants=network-online.target

 [Container]
 ContainerName=node-exporter
-Image=docker.io/prom/node-exporter:latest
+Image=docker.io/prom/node-exporter:latest@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
 Network=host
 PodmanArgs=--pid=host
 Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100
@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target

 [Container]
-Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329
+Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329@sha256:8f655bad7dcbc5398a78f70a5af12600b4ed3424f901566ed1eaa6710e9e2334
 ContainerName=plex
 Network=dankserver_appnet:alias=plex
 Environment=PUID=0
@@ -14,7 +14,7 @@ After=network.target

 [Container]
 ContainerName=romm-mariadb
-Image=docker.io/library/mariadb:11
+Image=docker.io/library/mariadb:11@sha256:f16d7a344413e519dff518f27bdb221388f84b0d274c97c4fb5f2cb0dc1b34fa
 Network=dankserver_appnet:alias=mariadb
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=MARIADB_DATABASE=romm
@@ -12,7 +12,7 @@ After=network.target

 [Container]
 ContainerName=romm-redis
-Image=docker.io/library/redis:7-alpine
+Image=docker.io/library/redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
 Network=dankserver_appnet:alias=redis
 Volume=/home/dank/Containers/romm/redis:/data
 HealthCmd=redis-cli ping
@@ -18,7 +18,7 @@ After=romm-mariadb.service romm-redis.service network.target

 [Container]
 ContainerName=romm
-Image=ghcr.io/rommapp/romm:latest
+Image=ghcr.io/rommapp/romm:latest@sha256:03295f95a6290e752641f9553ac4198b0e5b71b879d57f7b2371b98f608feb54
 Network=dankserver_appnet
 EnvironmentFile=/home/dank/Containers/romm/env
 Environment=DB_HOST=mariadb
@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target

 [Container]
-Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245
+Image=lscr.io/linuxserver/tautulli:v2.18.1-ls245@sha256:c66d3afa8f7a701a48dd1b1f57b22eca0483d895d670fe11ecff783f6a9bbd73
 ContainerName=tautulli
 Network=dankserver_appnet:alias=tautulli
 Environment=PUID=0
@@ -19,7 +19,7 @@ Description=Gluetun VPN tunnel (FrankBook-Pro podman machine)

 [Container]
 ContainerName=gluetun
-Image=docker.io/qmcgaw/gluetun:latest
+Image=docker.io/qmcgaw/gluetun:latest@sha256:c2ee4191762fca6b6a3c3359a731198a95d9c9af4db32d466e223bf3a84e779b
 AddCapability=NET_ADMIN
 AddDevice=/dev/net/tun:/dev/net/tun
 Environment=VPN_SERVICE_PROVIDER=${VPN_SERVICE_PROVIDER}
@@ -15,7 +15,7 @@ After=gluetun.service

 [Container]
 ContainerName=lidarr
-Image=lscr.io/linuxserver/lidarr:latest
+Image=lscr.io/linuxserver/lidarr:latest@sha256:d8549ff9ea29b4015c32257360652c1000c31ea471a2b3f0835ae77292faf35b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -24,7 +24,7 @@ After=gluetun.service

 [Container]
 ContainerName=prowlarr
-Image=lscr.io/linuxserver/prowlarr:latest
+Image=lscr.io/linuxserver/prowlarr:latest@sha256:b3b00047ed20f6ce2b16780145cb6a0acb51c9e28ea6aa6f5dcd33c7034f1797
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -29,7 +29,7 @@ Network=gluetun.container
 # Pinned to the last 4.x line: Readarr's retired qB client requires the
 # qB 4.x login response (HTTP 200, body "Ok."); qB 5.x answers 204 with an
 # empty body, which fails Readarr's auth check. All other clients accept 4.x.
-Image=lscr.io/linuxserver/qbittorrent:4.6.7
+Image=lscr.io/linuxserver/qbittorrent:4.6.7@sha256:55f15d44396315551f87294a176efae733b16e283f38980308e46073950257c6
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -15,7 +15,7 @@ After=gluetun.service

 [Container]
 ContainerName=radarr
-Image=lscr.io/linuxserver/radarr:latest
+Image=lscr.io/linuxserver/radarr:latest@sha256:fc0ae3c50f8128b97e84381903ec8d6845c3a569641cb7117c124376cccb57c1
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -19,7 +19,7 @@ After=gluetun.service

 [Container]
 ContainerName=readarr
-Image=lscr.io/linuxserver/readarr:amd64-1.0.980
+Image=lscr.io/linuxserver/readarr:amd64-1.0.980@sha256:a95e82dbb0eb2bb43de4c150f1dab9dd821306c67ea0b418983a957e731f2b61
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -16,7 +16,7 @@ After=gluetun.service

 [Container]
 ContainerName=sonarr
-Image=lscr.io/linuxserver/sonarr:latest
+Image=lscr.io/linuxserver/sonarr:latest@sha256:889dcc58c6aea859033985894b519b2e6b1e55d92527ed123f9f52e42ae7c97b
 Environment=PUID=0
 Environment=PGID=0
 Environment=TZ=America/Los_Angeles
@@ -34,7 +34,6 @@ SERVICE_ORDER = (
    "hermes",
    "hermes-dashboard",
    "hermes-webui",
-    "prometheus",
    "grafana",
    "loki",
    "alloy",
@@ -42,6 +41,9 @@ SERVICE_ORDER = (
    "cadvisor",
    "gpu-exporter",
    "alertmanager",
+    # The prober must answer before prometheus re-runs the services job.
+    "blackbox-exporter",
+    "prometheus",
    "authentik-postgres",
    "authentik-server",
    "authentik-worker",
@@ -97,6 +99,7 @@ DANKCLAW_SERVICES = {
    "cadvisor",
    "gpu-exporter",
    "alertmanager",
+    "blackbox-exporter",
    "authentik-postgres",
    "authentik-server",
    "authentik-worker",
@@ -277,6 +280,8 @@ def services_for_path(path: str, target: str) -> set[str]:
            # Provisioned datasources and dashboards: git is the source of
            # truth, so a change here must restart the service that reads them.
            _add(services, "grafana")
+        elif group == "blackbox":
+            _add(services, "blackbox-exporter")
        elif group == "homepage":
            # Homepage is a Dankclaw-owned deployment. Shared homepage files
            # are intentionally not copied during a Dankserver-only run.
@@ -380,7 +385,7 @@ def services_for_path(path: str, target: str) -> set[str]:
            # consumer (prometheus before grafana, loki before alloy).
            if len(parts) >= 4 and parts[2] == "quadlet":
                name = parts[3].removesuffix(".container")
-                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager"}:
+                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager", "blackbox-exporter"}:
                    _add(services, name)
            return services
        if len(parts) >= 2 and parts[1] == "gitea":
@@ -86,7 +86,7 @@ service_is_allowed() {
        dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
        dankclaw:comfyui|dankclaw:hermes|\
        dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|dankclaw:blackbox-exporter|\
        dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
        dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -252,16 +252,24 @@ install_signal_relay() {
    # The relay binds the host's LAN IP (the one address a bridge container
    # can reach as host.containers.internal), so keep the unauthenticated
    # endpoint off the rest of the LAN: admit loopback, the host itself
-    # (its own health probe) and the podman bridge subnet only. Idempotent:
-    # -C before -I.
+    # (its own health probe) and the podman bridge subnet only.
+    # ORDER IS THE GATE: `iptables -I` inserts at position 1, so the old
+    # shape (-C/-I x3 ACCEPT, then -C/-I the DROP) landed the blanket DROP
+    # ABOVE the ACCEPTs and every probe timed out beside a LISTENing socket
+    # (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT; the whole
+    # dankclaw drain aborted at the alertmanager gate twice before the cause
+    # was read off `nft list ruleset`). Rebuild the tail instead of inserting
+    # it: delete any blanket DROP wherever it sits, keep the three ACCEPTs
+    # at the head (-C before -I, unchanged), then APPEND the DROP so it is
+    # last and the ACCEPTs decide first. Converges on poisoned hosts.
+    run_privileged iptables -D INPUT -p tcp --dport 18093 -j DROP 2>/dev/null || true
    run_privileged iptables -C INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT 2>/dev/null \
        || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT
    run_privileged iptables -C INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT 2>/dev/null \
        || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT
    run_privileged iptables -C INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT 2>/dev/null \
        || run_privileged iptables -I INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT
-    run_privileged iptables -C INPUT -p tcp --dport 18093 -j DROP 2>/dev/null \
-        || run_privileged iptables -I INPUT -p tcp --dport 18093 -j DROP
+    run_privileged iptables -A INPUT -p tcp --dport 18093 -j DROP
    # The interpreter needs a moment to bind after restart; a one-shot curl
    # raced the socket on 2026-09-17 and failed a healthy deploy.
    local i
@@ -311,8 +319,45 @@ copy_loki_config() {
 }

 copy_alloy_config() {
-    run_privileged mkdir -p /opt/alloy/config
-    run_privileged cp shared/alloy/config.alloy /opt/alloy/config/config.alloy
+    # The repo config carries the token GRAFANA_HOST_LABEL; the installed copy
+    # carries the host's name, so Loki streams say WHICH machine's journal a
+    # line came from without a second config file per host.
+    # *_CONF_ROOT are test seams: the bats cases execute these copiers for
+    # real against a sandbox with a stubbed run_privileged.
+    local root="${ALLOY_CONF_ROOT:-/opt/alloy}" tmp
+    tmp="$(mktemp)"
+    sed "s/GRAFANA_HOST_LABEL/$TARGET/g" shared/alloy/config.alloy > "$tmp"
+    if ! grep -q "host = \"$TARGET\"," "$tmp" || grep -q "GRAFANA_HOST_LABEL" "$tmp"; then
+        echo "alloy config: host-label substitution failed" >&2
+        rm -f "$tmp"
+        return 1
+    fi
+    # Syntax gate: alloy's config language has no schema — a misplaced
+    # attribute (e.g. `on_error` outside the json stage) parses only at RUN
+    # time, and the unit then crash-loops with a healthy-looking deploy.
+    # Validate against the very image that will run it, before installing.
+    # Skipped only in the bats sandbox (no image there).
+    if [ "$root" = "/opt/alloy" ] || podman image exists localhost/franksdank/alloy:latest 2>/dev/null; then
+        if ! podman run --rm -v "$tmp:/c.alloy:ro" \
+                --entrypoint /usr/bin/alloy localhost/franksdank/alloy:latest \
+                validate /c.alloy; then
+            echo "alloy config: validate FAILED against the pinned image" >&2
+            rm -f "$tmp"
+            return 1
+        fi
+    fi
+    run_privileged mkdir -p "$root/config"
+    run_privileged cp "$tmp" "$root/config/config.alloy"
+    rm -f "$tmp"
+}
+
+copy_blackbox_config() {
+    # Single-file mount target: the unit binds this exact path into
+    # /etc/blackbox_exporter/config.yml. cp (not cp -r) keeps it one file.
+    # BLACKBOX_CONF_ROOT is the same test seam.
+    local root="${BLACKBOX_CONF_ROOT:-/opt/blackbox}"
+    run_privileged mkdir -p "$root"
+    run_privileged cp shared/blackbox/blackbox.yml "$root/blackbox.yml"
 }

 copy_grafana_provisioning() {
@@ -321,14 +366,25 @@ copy_grafana_provisioning() {
    # datasource would be invisible to review, lost with the data volume, and
    # unknown to anyone reading the repository. The unit already mounts
    # /opt/grafana/provisioning read-only, so this only has to populate it.
-    run_privileged mkdir -p /opt/grafana/provisioning/datasources \
-        /opt/grafana/provisioning/dashboards
+    # GRAFANA_PROV_ROOT is the test seam: point it at a sandbox and the whole
+    # copy runs unprivileged (the bats case executes this function for real).
+    local root="${GRAFANA_PROV_ROOT:-/opt/grafana}"
+    run_privileged mkdir -p "$root/provisioning/datasources" \
+        "$root/provisioning/dashboards"
    run_privileged cp shared/grafana/provisioning/datasources/prometheus.yml \
-        /opt/grafana/provisioning/datasources/prometheus.yml
+        "$root/provisioning/datasources/prometheus.yml"
    run_privileged cp shared/grafana/provisioning/dashboards/provider.yml \
-        /opt/grafana/provisioning/dashboards/provider.yml
-    run_privileged cp shared/grafana/provisioning/dashboards/llm-fleet.json \
-        /opt/grafana/provisioning/dashboards/llm-fleet.json
+        "$root/provisioning/dashboards/provider.yml"
+    # EVERY dashboard JSON rides, not a hand-listed subset: the fleet provider
+    # turns each file in the directory into a dashboard, so a copier that names
+    # files one by one silently drops new boards (measured 2026-10-21: the
+    # containers board existed in the repo since its commit and was NEVER
+    # provisioned — the panel-census browser check could not find it while the
+    # hand-placed older boards happened to be on disk). Same rule as the quadlet
+    # tier: the directory is the contract, the copy is generated state.
+    for dash in shared/grafana/provisioning/dashboards/*.json; do
+        run_privileged cp "$dash" "$root/provisioning/dashboards/$(basename "$dash")"
+    done
 }

 render_caddyfile() {
@@ -552,6 +608,9 @@ deploy_observability() {
            ;;
        cadvisor)
            ;;
+        blackbox-exporter)
+            copy_blackbox_config
+            ;;
        *)
            echo "unknown observability service: $service" >&2
            return 2
@@ -566,6 +625,9 @@ deploy_observability() {
        loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
        node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
        cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
+        # Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address
+        # is not bindable from the host netns — see the unit header).
+        blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
    esac
 }

@@ -2034,7 +2096,7 @@ deploy_service() {
        plex) deploy_plex ;;
        tautulli) deploy_tautulli ;;
        alloy) deploy_alloy ;;
-        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
+        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager|blackbox-exporter) deploy_observability "$service" ;;
        gpu-exporter) deploy_gpu_exporter ;;
        comfyui) deploy_comfyui ;;
        *) deploy_compose_service "$service" ;;
@@ -140,7 +140,7 @@ valid_service() {
        dankclaw:gitea-server|dankclaw:homepage|\
        dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
        dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
-        dankclaw:alertmanager|\
+        dankclaw:alertmanager|dankclaw:blackbox-exporter|\
        dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
        dankclaw:open-notebook|\
@@ -130,10 +130,19 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix

 ## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)

- **Hard-fail keys** — the whole unit silently fails to generate
-  (`systemctl --user cat <unit>` shows "could not be found"): any
-  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
-  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
+- **Hard-fail keys abort the WHOLE BATCH, not one unit** (podman 5.7.0,
+  live 2026-10-21): one unsupported key — e.g. `HostAddr=` — makes the
+  generator log `unsupported key … in group 'Container'` plus
+  `processing encountered some errors` and generate NOTHING for that
+  manager. The running containers keep going as orphans, so the failure
+  first appears on the next deploy as `Unit <x>.service not found` for a
+  service that is visibly up. Other offenders: any dash-prefixed
+  `[Container]` option (`--shm-size=…`), `Command=` (use `Exec=`).
+  The seam `tests/quadlet_image_pins.bats` fails the build on any
+  generator-rejected key in any host's units.
+  Also: an image whose ENTRYPOINT is the binary takes FLAGS ONLY in
+  `Exec=` — repeating the binary makes it a positional arg
+  (`unexpected /bin/blackbox_exporter`, live 2026-10-21).
 - **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
  spaces into the healthcheck argv; a quoted probe
  `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
@@ -148,6 +157,16 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
  "unsupported key"`, `systemctl --user cat probe.service` (shows the
  generated ExecStart verbatim), then remove + reload.

+## 4a. Image pins (fleet law, 2026-10-21)
+
+Every registry image in a quadlet is digest-pinned (`tag@sha256:…`);
+locally built images (`localhost/*`, `franksdank/*`) are exempt — their
+build is the pin. A rolling tag lets a rebuild move under a running
+service with nobody approving it, and it makes a CVE audit ungradeable.
+Gate: `tests/quadlet_image_pins.bats`. Where upstream publishes no version
+tags (copyparty-ac stops at 1.19.15), pin `:latest@sha256:` and name the
+version the config label carries in the comment beside it.
+
 ## 5. Rootless Quadlet log retrieval

 Step 0 — establish ownership: `systemctl --user status <name>.service`
@@ -110,9 +110,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file

 ## Recommended additions

-Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
-reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
-2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
+Quantized preferred. The iGPU's GTT aperture is not free RAM beside
+frank-dragoon — size every pick against *What actually fits beside
+frank-dragoon* below (the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
 all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
 `huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
 `huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
@@ -133,8 +133,34 @@ sha256 checks) or a plain `hf download` with the token in the environment —
 never on the command line.

 Skip list: the original `city96/Qwen-Image-gguf` conversions (superseded by
-the 2.1 files already on disk), and any BF16 of #2/#4 (the iGPU does not need
-them at GTT capacity, and they double the disk bill).
+the 2.1 files already on disk), and any BF16 of #2/#4 — they double the disk
+bill, and see *What actually fits beside frank-dragoon* for why raw GTT size
+is not the real budget.
+
+## What actually fits beside frank-dragoon
+
+The 122 GB `/system_stats` figure is the GTT **aperture**, not free memory: on
+an APU the aperture and `MemAvailable` are the same physical RAM, and
+frank-dragoon pins tens of GiB of it. Measured 2026-10-21 with frank-dragoon
+live: `MemAvailable` 15.3 GiB, iGPU busy 1.75 GiB — ComfyUI itself idles at
+~10 MB, so the budget for a generation run is roughly what `free` says.
+
+Weights must land inside that budget with slack for compute buffers and
+activations (the mmproj-class lesson: price a model at ~1.5× its file set):
+
+| Pipeline (weights) | Fits beside frank-dragoon? |
+|---|---|
+| Qwen-Image 2.1 int8_convrot + w4a8 + bf16 VAE (~14.2 GB) — the two proven workflows | **No** — thrash-or-fail; do not queue it while the endpoint serves |
+| GGUF Q8_0 + w4a8 + small VAE (~13.3 GB) | No, same reason |
+| Any BF16 combination | No |
+| SDXL-Turbo / SD 1.5-class (~2–7 GB) | Yes, with real headroom |
+
+**The law: one process owns the memory — same rule as the 3090 bench law.**
+For a Qwen-Image render: stop and mask frank-dragoon
+(`sudo systemctl stop frank-dragoon` + `mask` — the mask stops
+`Restart=always` from firing mid-render), generate, unmask, verify
+`curl 192.168.0.143:7770/health`. The proven workflows above then run with
+the whole machine. Do not treat `/system_stats`'s 122 GB as an invitation.

 ## See Also

@@ -129,6 +129,12 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
 - Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
 - The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
 - The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
+- **The 122 GB `/system_stats` VRAM figure is the GTT aperture, not free
+  memory.** On the APU it is system RAM, and frank-dragoon pins most of it
+  (`MemAvailable` 15.3 GiB with the endpoint live, measured 2026-10-21). A
+  Qwen-Image-class render beside the running endpoint is thrash-or-fail; the
+  stop-mask-render-verify procedure and the fit table are in
+  [[ComfyUI Qwen-Image Workflows]] → What actually fits beside frank-dragoon.

 ## See Also

@@ -34,15 +34,16 @@ https://github.com/open-webui/open-webui
 The picker is built from the `model` table in `webui.db` (one row per model)
 plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
 connection discovery — so the picker shows exactly what is seeded there.
-As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified
-2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids
-`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
-`shared/homepage/services.json`)*.
+Since 2026-10-21 the picker exposes **all three fleet models to every
+account**: `frank-dragoon`, `frank-archon`, `frank-zealot` (per Frank: open
+`frank-archon` to everyone; the other two were already public). Proven with
+a `role=user` token: `/api/models` lists all three ids and a chat turn to
+`frank-archon` answers end-to-end.

-| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
+| Connection (`openai.api_configs` idx) | `openai.api_base_urls` (live 2026-10-21) | Enabled | `model_ids` | Endpoint page |
 |---|---|---|---|---|
-| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
-| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
+| 0 | `http://host.containers.internal:7770/v1` | yes | `frank-dragoon` | [[frank-dragoon]] |
+| 1 | `http://192.168.0.235:7770/v1` | yes | `frank-archon` | [[frank-archon]] |
 | 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |

 The URLs in the first column are the DB's recorded values and predate the 2026-08-22
@@ -50,9 +51,10 @@ bridge move; on this network they are wrong — see the reachability gotcha belo
 is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
 that day's payload swap (see [[frank-zealot]] for what was deployed and when).

-The `model` table carries exactly two decoration rows, `frank-zealot` and
-`frank-dragoon`. Both the context length and the description live **inside
-JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
+The `model` table carries three decoration rows — `frank-zealot`,
+`frank-dragoon`, `frank-archon` (200,000 context, added 2026-10-21).
+Both the context length and the description live **inside JSON columns** —
+`params` holds `{"context": {"length": …}}` and `meta` holds
 `{"description": …}`; there is no `context_length` or `description` column, so
 `sqlite3` one-liners against those names fail. `params.context.length` is a copy
 of the endpoint's per-slot window, so the authoritative number lives on the
@@ -380,6 +382,20 @@ neither.
  name: /selected model/i })`) and the welcome text "How can I help you
  today?"; that snapshot is also the cheapest proof the picker is populated.

+- **2026-10-21 — picker opened to all three models, for everyone.** Per
+  Frank: `frank-archon` accessible to everyone. Three-leg repair on
+  `webui.db` (snapshot `webui.db.bak-20261021` first): connection idx 1
+  `enable: true`, a `model` row `frank-archon` (params
+  `{"context":{"length":200000}}`, meta description from the endpoint card),
+  and an `access_grant` row `(resource_type='model', resource_id='frank-archon',
+  principal_id='*', 'read')` — `principal_id='*'` is the public grant, not a
+  role row. Verified as a plain `role=user` token: `/api/models` lists all
+  three ids, chat to `frank-archon` answers. Token-mint gotchas: the backend
+  package imports only with cwd `/app/backend`, and `WEBUI_SECRET_KEY` must
+  come from **PID 1's environ** (`podman exec` sessions do not inherit the
+  app process env; the `.webui_secret_key` file happens to equal it — use it
+  explicitly, `podman exec -e`).
+
 ## See Also

 - [[Dankclaw Stack]]
@@ -20,6 +20,8 @@ committed.
 | `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
 | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
 | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
+| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`); the listen bind lives on the unit's `Exec` |

 Targets and datasources reference **container names** (`grafana:3000`
 `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
 reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
 the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
 (`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
-HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
-[[Stale Signal Audit]]).
+HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
+deployed 2026-10-21; see [[Stale Signal Audit]]).

 ## The deploy path

@@ -43,8 +45,9 @@ from `deploy_observability`:
 - `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
 - `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
 - `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
+- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
+- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)

 `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
 `alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
@@ -82,11 +85,14 @@ reports 4 alerting + 2 recording. Rules reload with the prometheus service.
 the backend (`http://prometheus:9090`), not loopback.

 **Add a dashboard** — drop the JSON into
-`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
-unified storage, not (only) the SQLite table — grep the API, not the DB, when
-verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
-2026-10-18: the storage-backend claim needs an admin password to confirm
-against `grafana.db` and the API side by side)*
+`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
+the next `--service grafana` deploy provisions it. This is a repaired law, not
+a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
+`containers` board — merged, tested, documented as "working" — had **never
+been provisioned**; the other boards existed on disk only because someone had
+hand-copied them once. A repo JSON is not a live dashboard until the copier
+carries it, and `deploy.sh` cannot see the gap because it only reports the
+copy it performed.

 ## Verify through the APIs, not the filesystem

@@ -95,17 +101,22 @@ against `grafana.db` and the API side by side)*
 curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
 curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
-# grafana: datasource + dashboard provisioned
-curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
+# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
+# account is the Authentik RP login). The proven surface is a real browser
+# session: warm the Anubis challenge on the vhost root, then hit
+# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
+# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
 ```

 The two Prometheus queries were run on 2026-10-18 and are what every number on
 this page comes from: 4 alerting + 2 recording rules, and `up` for
 `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
-`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
-listed in [[Prometheus]]. The Grafana query needs the local admin password and
-was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
-proof of installation only — see the next paragraph.
+`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
+that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
+2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
+filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
+only — see *Add a dashboard* for why a repo JSON is not a live board.

 Health answers from a stale process and provisioned resources can hide in
 unified storage — a file existing on disk is not proof the running server
@@ -74,11 +74,19 @@ published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
 — while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
 every same-host Prometheus target uses the gateway and only cross-host targets
 (dankserver) use plain LAN. Whether LAN works at all is network-specific: the
-relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
-iptables INPUT admitting loopback, the host itself and the podman bridge subnet
-`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
-point a scrape target at a host LAN IP from its own host on the strength of
-one working example; dial the gateway and probe.
+relay at `192.168.0.143:18093` is reached from alertmanager's network only
+while `install_signal_relay`'s INPUT rules are in force: loopback, the host
+itself and the podman bridge subnet `10.89.0.0/16` accepted **above** a
+trailing blanket DROP. The law the poisoned host taught (2026-10-21):
+`iptables -I` prepends, so inserting the DROP last lands it first — the old
+install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the
+deployer's health gate failed beside a LISTENing socket, and the drain
+aborted twice before `nft list ruleset` named the cause. The install now
+rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head,
+DROP appended); verify any change to it with the sandboxed bats case, not
+with `-C` greps — `-C` answers existence, never order. Never point a scrape
+target at a host LAN IP from its own host on the strength of one working
+example; dial the gateway and probe.

 ## Deployment

@@ -95,13 +103,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
 [[Observability Provisioning]] for what is provisioned as code and how to
 verify it through the APIs.

-## Current state (2026-10-20)
+## Current state (2026-10-21)

 * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
-  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
-  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
-  and both node exporters. The four downs are structural (the `comfyui` and
-  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
+  every job up — the three LLM endpoints (frank-archon on `vllm:*`,
+  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
+  both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
+  the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
+  downs are retired — see [[Prometheus]] → Targets.
  Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
  (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
  alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
@@ -109,12 +118,15 @@ verify it through the APIs.
 * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
  relay, and `journalctl --user -u signal-relay` records successful
  `sent to +12062476336:` lines for every rule in the set.
-* **Dashboards: working.** Provisioned datasources — Prometheus (uid
-  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
-  `containers`, `observability` and `logs-overview` dashboards, all
-  installed from `shared/grafana/provisioning/`. The Loki datasource is how
-  you read logs: Loki has no web UI of its own, so Grafana Explore is the
-  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
+* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
+  datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
+  `llm-fleet`, `host-system`, `containers`, `observability` and
+  `logs-overview` boards. Until 2026-10-21 the copier carried only
+  `llm-fleet.json` by name and the `containers` board had never been
+  provisioned; the deployer now globs the directory. The Loki datasource is
+  how you read logs: Loki has no web UI of its own, so Grafana Explore is the
+  human surface (the guarded `loki.franksdank.cloud` vhost serves only the
+  API). Verify method and the copier law: [[Observability Provisioning]].
 * **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
  after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
  per-container CPU/memory here, and on this rootless-Podman host it works on one
@@ -139,26 +151,26 @@ verify it through the APIs.
  `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
  the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
  node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
-* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
-  a rootless Alloy cannot read the `0640` journal files, and the container-log
-  globs point at a root-storage layout that does not exist here. The fix is built
-  and tested in-repo (2026-10-20): Alloy is now a **root system service on
-  `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
-  Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
-  lines and pushed 26,966 with zero drops. The **live** instance is still the old
-  rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
-  [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
-* **Not configured:** no blackbox exporter (the structural downs have no real
-  uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
-  not yet deployed (see *Logs* above).
+* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
+  pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
+  the container-log globs pointed at a root-storage layout that does not exist
+  here) — fixed 2026-10-20 by making Alloy a **root system service on
+  `Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
+  but the stream carried only `job`/`service_name`, so every per-container
+  view was structurally empty: Loki metrics queries group only by INDEXED
+  stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
+  label (plus a `host` label) at ingestion — `loki.process` in
+  `shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
+  purpose-test law in [[Observability Provisioning]].

 ## Citations

- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
+- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
 - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
 - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
 - `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
 - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
 - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
+- `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
 - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
 - Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
@@ -42,29 +42,29 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —

 The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.

-**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
+**No job reads down structurally any more** (the three permanent-reds were
+retired 2026-10-21; census re-run after each deploy):

-| Job | Target | `health` | `lastError`, verbatim |
-|-----|--------|----------|------------------------|
-| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
-| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
-| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
+| Job | Target | how |
+|---|---|---|
+| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
+| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |

-Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
+Scraping a web root is not a health check: an HTML page fails the scrape
+protocol (`unsupported Content-Type`), which is what kept those targets red
+beside healthy services. The classification rule and the honesty law live in
+[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
+netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on loopback and
+the LAN, and Prometheus scrapes it at `host.containers.internal` — traffic to
+that gateway lands on the host's loopback listener. The gateway address is
+not itself bindable from the host netns.
+Plain targets stay `host:port`, never URLs — the probe job's URL targets are
+the documented exception, relabeled onto the prober.

 **`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.

 Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.

-The remedy for a probe-shaped question is the **blackbox exporter**; the
-classification rule and the honesty law for it live in [[Stale Signal Audit]].
-Until the exporter is deployed, these three stay red for a structural reason:
-deleting them would
-hide a real service, and fixing them by publishing databases or enabling metrics
-would trade a wrong signal for a real exposure. Targets are `host:port`, never
-URLs.
-
 ## Volumes

 | Container path | Host path |
@@ -17,6 +17,9 @@ logging {
 // Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
 // host network, so it pushes by loopback, never by container name — a root
 // container is not on the rootless dankclaw_default network.
+// GRAFANA_HOST_LABEL is replaced with this host's name by
+// copy_alloy_config at deploy time; it must not appear verbatim in the
+// installed config.
 loki.write "default" {
  endpoint {
    url = "http://127.0.0.1:3100/loki/api/v1/push"
@@ -33,6 +36,33 @@ loki.source.journal "journal" {
  format_as_json = true
  labels = {
    job  = "systemd-journal",
+    host = "GRAFANA_HOST_LABEL",
+  }
+  forward_to = [loki.process.relabel.receiver]
+}
+
+// Promote the fields that ARE the point of the log stream to real Loki labels.
+// Without this the journal arrives as one opaque JSON line per entry and every
+// per-container view is structurally impossible: Loki metrics queries can only
+// group by INDEXED stream labels, so `sum by (container_name)` returns a single
+// anonymous series and the logs-overview board read empty beside 6.5k
+// lines/5min of live traffic (measured 2026-10-21). CONTAINER_NAME is what
+// podman's journald logging writes (uppercase); host units lack it and stay on
+// the host-only stream. `format_as_json = true` above guarantees every line
+// reaching this stage is a JSON object, so there is no non-JSON branch to
+// protect — alloy 1.21's json stage has no `on_error` attribute (validated
+// against the pinned image); a malformed line would drop, which is the
+// correct fate for a line Loki could not index anyway.
+loki.process "relabel" {
+  stage.json {
+    expressions = {
+      container = "CONTAINER_NAME",
+    }
+  }
+  stage.labels {
+    values = {
+      container = "container",
+    }
  }
  forward_to = [loki.write.default.receiver]
 }
@@ -0,0 +1,17 @@
+# Blackbox exporter modules — the honest answer to "is this HTTP service
+# answering?". A Prometheus scrape cannot express it (an HTML front page is a
+# 200 that fails the scrape protocol), which is why the old `services` and
+# `comfyui` jobs sat permanently red on `unsupported Content-Type`.
+# See serverAppConfigs-docs/Architecture/Stale Signal Audit.md §1.
+# The listen address is NOT set here: the quadlet unit's Exec passes
+# --listen-address (LAN bind — see the unit header for why loopback is
+# unreachable from the bridge-networked Prometheus). Keep this file to
+# modules only, so the two sources cannot disagree.
+
+modules:
+  http_2xx:
+    prober: http
+    timeout: 10s
+    http:
+      preferred_ip_protocol: ip4
+      valid_status_codes: [] # default: 2xx + 3xx
@@ -2,74 +2,242 @@
  "uid": "logs-overview",
  "title": "Logs \u2014 Loki (systemd journal)",
  "description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
-  "tags": ["logs", "loki"],
+  "tags": [
+    "logs",
+    "loki"
+  ],
  "timezone": "browser",
  "schemaVersion": 39,
  "refresh": "30s",
-  "time": { "from": "now-1h", "to": "now" },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
  "templating": {
    "list": [
      {
        "name": "container",
        "label": "container",
        "type": "query",
-        "datasource": { "type": "loki", "uid": "loki" },
-        "query": { "query": "{job=\"systemd-journal\"} | json", "label": "container_name", "refId": "A" },
+        "datasource": {
+          "type": "loki",
+          "uid": "loki"
+        },
+        "query": {
+          "query": "label_values({job=\"systemd-journal\"}, container)",
+          "refId": "LokiVariableQueryEditor-VariableQuery"
+        },
        "includeAll": true,
        "multi": true,
        "refresh": 2,
-        "current": { "text": "All", "value": "$__all" }
+        "current": {
+          "text": "All",
+          "value": "$__all"
+        }
      }
    ]
  },
  "panels": [
    {
-      "id": 1, "type": "timeseries", "title": "Log volume by container (lines/s)",
-      "description": "Parse the journal JSON, count lines per container. Spikes = a chatty or crashing unit.",
-      "gridPos": { "h": 9, "w": 24, "x": 0, "y": 0 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps", "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] },
-      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+      "id": 1,
+      "type": "timeseries",
+      "title": "Log volume by container (lines/s)",
+      "description": "Per-container journal line rate. The `container` label is promoted at ingestion by Alloy (loki.process) from podman's CONTAINER_NAME journald field - Loki metrics can only group by indexed stream labels.",
+      "gridPos": {
+        "h": 9,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "custom": {
+            "fillOpacity": 20,
+            "stacking": {
+              "mode": "normal"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 2, "type": "timeseries", "title": "Total log rate",
+      "id": 2,
+      "type": "timeseries",
+      "title": "Total log rate",
      "description": "All journal lines per second.",
-      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 9 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
-      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))", "legendFormat": "all lines/s", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))",
+          "legendFormat": "all lines/s",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 3, "type": "timeseries", "title": "Error-level line rate",
+      "id": 3,
+      "type": "timeseries",
+      "title": "Error-ish line rate (text match)",
      "description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
-      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 9 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "fixed", "fixedColor": "red" } }, "overrides": [] },
-      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))", "legendFormat": "error lines/s", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 9
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "color": {
+            "mode": "fixed",
+            "fixedColor": "red"
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))",
+          "legendFormat": "error lines/s",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 4, "type": "timeseries", "title": "Errors by container",
+      "id": 4,
+      "type": "timeseries",
+      "title": "Errors by container",
      "description": "Which unit is throwing. Rank by the top offenders.",
-      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 17 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
-      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (container) (count_over_time({job=\"systemd-journal\", container!=\"\"} |~ \"(?i)(error|panic|fatal)\" [$__auto]))",
+          "legendFormat": "{{container_name}}",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 5, "type": "stat", "title": "Error lines (last 15m)",
+      "id": 5,
+      "type": "stat",
+      "title": "Error lines (last 15m)",
      "description": "Absolute count over the window, not a rate.",
-      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 17 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 50 }, { "color": "red", "value": 500 }] } }, "overrides": [] },
-      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))", "instant": true, "legendFormat": "errors/15m", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 17
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
              },
              {
-      "id": 6, "type": "logs", "title": "Journal (filtered)",
+                "color": "yellow",
+                "value": 50
+              },
+              {
+                "color": "red",
+                "value": 500
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))",
+          "instant": true,
+          "legendFormat": "errors/15m",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "logs",
+      "title": "Journal (filtered)",
      "description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
-      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 25 },
-      "datasource": { "type": "loki", "uid": "loki" },
-      "options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
-      "targets": [{ "expr": "{job=\"systemd-journal\"} | json | container_name=~\"$container\"", "refId": "A" }]
+      "gridPos": {
+        "h": 12,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "datasource": {
+        "type": "loki",
+        "uid": "loki"
+      },
+      "options": {
+        "showTime": true,
+        "wrapLogMessage": true,
+        "sortOrder": "Descending",
+        "enableLogDetails": true
+      },
+      "targets": [
+        {
+          "expr": "{job=\"systemd-journal\", container=~\"$container\"}",
+          "refId": "A"
+        }
+      ]
    }
  ]
 }
@@ -2,95 +2,384 @@
  "uid": "observability",
  "title": "Observability \u2014 self-monitoring",
  "description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
-  "tags": ["observability", "meta"],
+  "tags": [
+    "observability",
+    "meta"
+  ],
  "timezone": "browser",
  "schemaVersion": 39,
  "refresh": "30s",
-  "time": { "from": "now-6h", "to": "now" },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
  "panels": [
    {
-      "id": 1, "type": "stat", "title": "Targets up",
-      "gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] } }, "overrides": [] },
-      "targets": [{ "expr": "count(up == 1)", "instant": true, "refId": "A" }]
+      "id": 1,
+      "type": "stat",
+      "title": "Targets up",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 0,
+        "y": 0
      },
-    {
-      "id": 2, "type": "stat", "title": "Targets down",
-      "description": "Known structural downs: comfyui (no metrics endpoint), gitea (metrics off), and the homepage 'services' probes. Real regressions ride up from here too \u2014 check the table before dismissing.",
-      "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 1 }] } }, "overrides": [] },
-      "targets": [{ "expr": "count(up == 0)", "instant": true, "refId": "A" }]
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
              {
-      "id": 3, "type": "stat", "title": "TSDB head series",
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 1)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 2,
+      "type": "stat",
+      "title": "Targets down",
+      "description": "Every scrape target that is down. ZERO IS THE HEALTHY STATE: the HTML-probe jobs and the disabled Gitea metrics were retired 2026-10-21 (HTTP liveness now rides the blackbox exporter, job \"services\"). A non-zero count here is a real regression - read /api/v1/targets lastError before triaging.",
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 4,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "count(up == 0)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "stat",
+      "title": "TSDB head series",
      "description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
-      "gridPos": { "h": 4, "w": 5, "x": 8, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
-      "targets": [{ "expr": "prometheus_tsdb_head_series", "instant": true, "refId": "A" }]
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 8,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_tsdb_head_series",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "stat",
+      "title": "Discovered targets",
+      "gridPos": {
+        "h": 4,
+        "w": 5,
+        "x": 13,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "targets": [
        {
-      "id": 4, "type": "stat", "title": "Discovered targets",
-      "gridPos": { "h": 4, "w": 5, "x": 13, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
-      "targets": [{ "expr": "sum(prometheus_sd_discovered_targets)", "instant": true, "refId": "A" }]
+          "expr": "sum(prometheus_sd_discovered_targets)",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 5, "type": "stat", "title": "Prometheus mem",
-      "gridPos": { "h": 4, "w": 3, "x": 18, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
-      "targets": [{ "expr": "process_resident_memory_bytes{job=\"prometheus\"}", "instant": true, "refId": "A" }]
+      "id": 5,
+      "type": "stat",
+      "title": "Prometheus mem",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "process_resident_memory_bytes{job=\"prometheus\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "stat",
+      "title": "Grafana up",
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 0
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
            {
-      "id": 6, "type": "stat", "title": "Grafana up",
-      "gridPos": { "h": 4, "w": 3, "x": 21, "y": 0 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "mappings": [{ "type": "value", "options": { "1": { "text": "UP", "color": "green" }, "0": { "text": "DOWN", "color": "red" } } }], "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] } }, "overrides": [] },
-      "targets": [{ "expr": "up{job=\"grafana\"}", "instant": true, "refId": "A" }]
+              "type": "value",
+              "options": {
+                "1": {
+                  "text": "UP",
+                  "color": "green"
                },
+                "0": {
+                  "text": "DOWN",
+                  "color": "red"
+                }
+              }
+            }
+          ],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
              {
-      "id": 7, "type": "table", "title": "Target health",
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "up{job=\"grafana\"}",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "type": "table",
+      "title": "Target health",
      "description": "Every scrape target and its current up. Sort by Value to surface regressions.",
-      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 4 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "options": { "sortBy": [{ "displayName": "Value", "sortOrder": "Asc" }] },
-      "targets": [{ "expr": "up", "format": "table", "instant": true, "refId": "A" }]
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 0,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
      },
+      "options": {
+        "sortBy": [
          {
-      "id": 8, "type": "timeseries", "title": "Scrape interval drift (p99 actual)",
+            "displayName": "Value",
+            "sortOrder": "Asc"
+          }
+        ]
+      },
+      "targets": [
+        {
+          "expr": "up",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 8,
+      "type": "timeseries",
+      "title": "Scrape interval drift (p99 actual)",
      "description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
-      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 4 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
-      "targets": [{ "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}", "legendFormat": "{{instance}}", "refId": "A" }]
+      "gridPos": {
+        "h": 9,
+        "w": 12,
+        "x": 12,
+        "y": 4
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ]
    },
    {
-      "id": 9, "type": "timeseries", "title": "Scrape samples/s",
+      "id": 9,
+      "type": "timeseries",
+      "title": "Scrape samples/s",
      "description": "Total samples ingested across all jobs.",
-      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
-      "targets": [{ "expr": "sum(rate(scrape_samples_scraped[5m]))", "legendFormat": "samples/s", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short"
+        },
+        "overrides": []
      },
+      "targets": [
        {
-      "id": 10, "type": "timeseries", "title": "Loki request rate",
+          "expr": "sum(rate(scrape_samples_scraped[5m]))",
+          "legendFormat": "samples/s",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 10,
+      "type": "timeseries",
+      "title": "Loki request rate",
      "description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
-      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
-      "targets": [{ "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))", "legendFormat": "{{route}}", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 13
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps"
        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))",
+          "legendFormat": "{{route}}",
+          "refId": "A"
+        }
+      ]
+    },
    {
-      "id": 11, "type": "timeseries", "title": "Exporter process CPU",
+      "id": 11,
+      "type": "timeseries",
+      "title": "Exporter process CPU",
      "description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
-      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
-      "datasource": { "type": "prometheus", "uid": "prometheus" },
-      "fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
-      "targets": [{ "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))", "legendFormat": "{{job}}", "refId": "A" }]
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 21
+      },
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))",
+          "legendFormat": "{{job}}",
+          "refId": "A"
+        }
+      ]
    }
  ]
 }
@@ -60,20 +60,33 @@ scrape_configs:
      - targets: ["loki:3100"]
    metrics_path: "/metrics"

-  # ComfyUI (no native /metrics; probe the UI)
-  - job_name: "comfyui"
-    metrics_path: "/"
-    static_configs:
-      - targets: ["comfyui:8188"]
-
-  # Blackbox-style probes for service up/down (published LAN ports;
-  # gitea and comfyui are covered by their own jobs above)
+  # HTTP liveness, probed honestly through the blackbox exporter. Scraping a
+  # web root is NOT a health check — an HTML front page fails the scrape
+  # protocol (`unsupported Content-Type`), which is exactly why the old
+  # `services` and `comfyui` jobs were red forever beside healthy services
+  # (Stale Signal Audit §1). The prober is a host-netns container listening on
+  # the podman gateway (the scrape address below - the netavark hairpin gap
+  # bars this container from its own host's LAN IP, cadvisor job note) and on
+  # the LAN; from that netns it reaches the host LAN IP, and published ports
+  # via 127.0.0.1 - bridge container names do NOT resolve in the host netns,
+  # which is why comfyui's probe is a loopback URL. comfyui's own
+  # metrics-scrape job is retired, one owner per service.
  - job_name: "services"
-    metrics_path: "/"
+    metrics_path: /probe
+    params:
+      module: [http_2xx]
    static_configs:
      - targets:
-          - "192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
-          - "192.168.0.143:3005"   # OpenWebUI
+          - "http://192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
+          - "http://192.168.0.143:3005"   # OpenWebUI
+          - "http://127.0.0.1:8188"        # ComfyUI UI (published 0.0.0.0:8188; probed from the prober's host netns)
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: "host.containers.internal:9115"

  # LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
  # this container, not the guarded vhosts: these are machine clients and
@@ -193,7 +193,18 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
 }

@test "prometheus targets are host:port (no URL schemes)" {
-    ! grep -nE '^\s*-\s*"?http' "$REPO_ROOT/shared/prometheus/prometheus.yml"
+    # Plain scrape targets are host:port; a URL target is only meaningful for
+    # the blackbox prober job, whose relabel_configs copy it to __param_target
+    # and point __address__ at the prober. Anywhere else a URL target is the
+    # old HTML-probe defect wearing a new hat.
+    awk '
+        /job_name:/ { job=$0; sub(/.*"/, "", job); sub(/".*/, "", job); probe = 0 }
+        /metrics_path: \/probe/ { probe = 1 }
+        /^[[:space:]]*-[[:space:]]*"http/ && !probe {
+            print FILENAME ":" FNR ": URL target outside a prober job: " $0; bad = 1
+        }
+        END { exit bad }
+    ' "$REPO_ROOT/shared/prometheus/prometheus.yml"
 }

@test "observability containers run as container root (rootless = host user)" {
@@ -150,3 +150,23 @@ load helpers
        grep -q "$pair" "$BATS_TEST_TMPDIR/cli" || { echo "client gate missing $pair"; return 1; }
    done
 }
+
+@test "every gated dankclaw service is dispatched" {
+    # A service can clear both allowlists and still die inside the deployer:
+    # dankclaw has no compose file, and deploy_compose_service (the case
+    # catch-all) hard-fails with "no quadlet handler" since 2026-09-16. So
+    # every name the remote gate allows for dankclaw MUST be named by a
+    # deploy_service pattern — blackbox-exporter (2026-10-21) shipped with
+    # both gates and the health gate wired but no dispatch case, and only
+    # the deploy attempt found it.
+    run env OUT="$BATS_TEST_TMPDIR" bash -c '
+        grep -oE "dankclaw:[a-z0-9-]+" \
+            <(sed -n "/^service_is_allowed()/,/^}/p" "$1/scripts/deploy-remote.sh") \
+            | cut -d: -f2 | sort -u > "$OUT/gated"
+        sed -n "/^deploy_service()/,/^}/p" "$1/scripts/deploy-remote.sh" \
+            | grep -oE "^ +[a-z][a-z0-9|-]*\)" | tr -d " )" | tr "|" "\n" \
+            | sort -u > "$OUT/dispatched"
+        comm -23 "$OUT/gated" "$OUT/dispatched"
+    ' _ "$REPO_ROOT"
+    [ -z "$output" ] || { echo "gated dankclaw services with no deploy_service case:"; echo "$output"; return 1; }
+}
@@ -77,6 +77,7 @@ FRANKBOOK_UNITS=(
 }

@test "readarr keeps its pinned image" {
-    grep -q '^Image=lscr.io/linuxserver/readarr:amd64-1.0.980$' \
+    # 2026-10-21 fleet law: version tag AND digest (tests/quadlet_image_pins.bats)
+    grep -qE '^Image=lscr\.io/linuxserver/readarr:amd64-1\.0\.980@sha256:[0-9a-f]{64}$' \
        "$REPO_ROOT/frankbook-pro/readarr/quadlet/readarr.container" || return 1
 }
@@ -21,7 +21,7 @@ load helpers

 UNIT_DIR="$REPO_ROOT/dankclaw/observability/quadlet"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
-UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager)
+UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager blackbox-exporter)

 # container_section <unit> — the [Container] key/value block only
 container_section() {
@@ -33,7 +33,7 @@ service_section() {
    awk '/^\[Service\]/{f=1;next} /^\[/{f=0} f' "$1"
 }

-@test "all seven observability units exist under dankclaw/observability/quadlet/" {
+@test "all eight observability units exist under dankclaw/observability/quadlet/" {
    local svc
    for svc in "${UNITS[@]}"; do
        [ -f "$UNIT_DIR/$svc.container" ] || { echo "missing $svc.container"; return 1; }
@@ -559,6 +559,64 @@ assert any('ALERTS' in e for e in exprs), exprs
    grep -q -- '--dport 18093 -j DROP' "$handler"
 }

+@test "the deployer installs the relay firewall with the ACCEPTs deciding before the DROP" {
+    # Behavioral, run for real against a sandbox that plays the kernel: the
+    # rule set is a LIST and iptables -I prepends, so the shape this function
+    # had until 2026-10-21 (-C/-I ACCEPT x3, then -C/-I the blanket DROP)
+    # landed the DROP at position 1 and every packet died there — the drain
+    # failed its health gate beside a LISTENing socket twice. Seeding the
+    # sandbox with that poisoned order also pins the convergent repair: a
+    # re-deploy must rewrite the tail, not skip because -C finds every rule.
+    local sandbox="$BATS_TEST_TMPDIR/fw"
+    mkdir -p "$sandbox/bin"
+    cat > "$sandbox/bin/iptables" <<'EOF'
+#!/usr/bin/env bash
+# ordered rule list in $FW_STATE: -C lookup, -I at head, -A at tail, -D delete.
+state="${FW_STATE:?}"
+touch "$state"
+op="$1"; shift
+case "$op" in
+    -C) grep -Fxq -- "$*" "$state" ;;
+    -I) grep -Fxq -- "$*" "$state" || { { echo "$*"; cat "$state"; } > "$state.new"; mv "$state.new" "$state"; } ;;
+    -A) grep -Fxq -- "$*" "$state" || echo "$*" >> "$state" ;;
+    -D) if grep -Fxq -- "$*" "$state"; then
+            grep -Fxv -- "$*" "$state" > "$state.new"; mv "$state.new" "$state"
+        else exit 1; fi ;;
+    *) exit 0 ;;
+esac
+EOF
+    chmod +x "$sandbox/bin/iptables"
+    # Poisoned state exactly as the pre-fix deployer left Dankclaw.
+    printf '%s\n' \
+        'INPUT -p tcp --dport 18093 -j DROP' \
+        'INPUT -p tcp --dport 18093 -s 10.89.0.0/16 -j ACCEPT' \
+        'INPUT -p tcp --dport 18093 -s 192.168.0.143 -j ACCEPT' \
+        'INPUT -p tcp --dport 18093 -s 127.0.0.1 -j ACCEPT' \
+        > "$sandbox/rules"
+    (
+        cd "$REPO_ROOT"
+        export PATH="$sandbox/bin:$PATH" FW_STATE="$sandbox/rules" HOME="$sandbox/home"
+        mkdir -p "$HOME"
+        eval "$(sed -n '/^install_signal_relay()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        systemctl() { :; }
+        curl() { return 0; }
+        sleep() { :; }
+        install_signal_relay
+    )
+    # Exactly one blanket DROP, and it sits after every ACCEPT.
+    [ "$(grep -cF -- 'INPUT -p tcp --dport 18093 -j DROP' "$sandbox/rules")" -eq 1 ]
+    local drop_at total
+    drop_at="$(grep -nF -- 'INPUT -p tcp --dport 18093 -j DROP' "$sandbox/rules" | cut -d: -f1)"
+    total="$(wc -l < "$sandbox/rules")"
+    [ "$drop_at" -eq "$total" ]
+    local s seen
+    for s in 127.0.0.1 192.168.0.143 10.89.0.0/16; do
+        seen="$(grep -nF -- "-s $s -j ACCEPT" "$sandbox/rules" | cut -d: -f1)"
+        [ -n "$seen" ] && [ "$seen" -lt "$drop_at" ]
+    done
+}
+
@test "the planner selects alertmanager for its config and relay files" {
    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
        --paths shared/prometheus/alertmanager.yml
@@ -644,3 +702,125 @@ assert not any('container_network' in e or 'container_fs_' in e for e in names),
    [ "$status" -eq 0 ]
    [ "$output" = "cadvisor" ]
 }
+
+@test "the grafana copier provisions EVERY dashboard json, not a hand-listed subset" {
+    # Behavioral, not a text grep: the copier is extracted and executed with a
+    # stubbed run_privileged against a sandbox. The fleet provider turns each
+    # file in its directory into a dashboard, so a file the copier skips is a
+    # dashboard that silently never exists (the containers board lived in the
+    # repo, never reached /opt, and its fixes were undeployable).
+    # BATS_TEST_TMPDIR: per-test, removed by bats itself; a mktemp dir with a
+    # RETURN trap races bats' own trap machinery and vanishes mid-function.
+    local sandbox="$BATS_TEST_TMPDIR/sandbox"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_grafana_provisioning()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        export GRAFANA_PROV_ROOT="$sandbox/opt/grafana"
+        copy_grafana_provisioning
+    )
+    local f
+    for f in "$REPO_ROOT"/shared/grafana/provisioning/dashboards/*.json; do
+        [ -f "$sandbox/opt/grafana/provisioning/dashboards/$(basename "$f")" ]
+    done
+    [ -f "$sandbox/opt/grafana/provisioning/dashboards/provider.yml" ]
+    [ -f "$sandbox/opt/grafana/provisioning/datasources/prometheus.yml" ]
+}
+
+@test "every observability unit declares an Image" {
+    # Quadlet refuses to GENERATE a service for a .container without Image=
+    # - the deployer then dies at `systemctl restart` with "Unit not found"
+    # and no diagnosis (blackbox-exporter, live 2026-10-21: merged with the
+    # copier, gates and health check wired, but no Image line).
+    local svc
+    for svc in "${UNITS[@]}"; do
+        grep -q '^Image=' "$UNIT_DIR/$svc.container" \
+            || { echo "$svc.container has no Image= - quadlet will not generate it"; return 1; }
+    done
+}
+
+@test "changes under shared/blackbox/ plan the prober" {
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
+        --paths shared/blackbox/blackbox.yml
+    [ "$status" -eq 0 ]
+    [ "$output" = "blackbox-exporter" ]
+}
+
+@test "the copier runs the same code path the deployer runs" {
+    # Textually pinned so a refactor cannot quietly route the deployer around
+    # the copy_* functions the seams execute.
+    grep -q '        prometheus)' "$REMOTE_DEPLOY"
+    grep -q 'copy_prometheus_config' "$REMOTE_DEPLOY"
+    grep -q 'copy_alertmanager_config' "$REMOTE_DEPLOY"
+    grep -q 'copy_loki_config' "$REMOTE_DEPLOY"
+    grep -q '        blackbox-exporter)' "$REMOTE_DEPLOY"
+    grep -q 'copy_blackbox_config' "$REMOTE_DEPLOY"
+}
+
+@test "the alloy copier installs the host-labeled config, token-free" {
+    # Executes copy_alloy_config for real against a sandbox (stubbed
+    # run_privileged). A surviving GRAFANA_HOST_LABEL token would ship a
+    # config whose host label is a literal placeholder — the exact class of
+    # silent-label defect this tranche retired the board for.
+    local sandbox="$BATS_TEST_TMPDIR/opt"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        # NOTE: VAR=x func does not bind for shell functions — export.
+        export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
+        copy_alloy_config
+    )
+    [ -f "$sandbox/config/config.alloy" ]
+    grep -q 'host = "dankclaw",' "$sandbox/config/config.alloy"
+    ! grep -q "GRAFANA_HOST_LABEL" "$sandbox/config/config.alloy"
+}
+
+@test "the blackbox copier installs the prober config" {
+    local sandbox="$BATS_TEST_TMPDIR/blackbox"
+    (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_blackbox_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        export BLACKBOX_CONF_ROOT="$sandbox"
+        copy_blackbox_config
+    )
+    [ -f "$sandbox/blackbox.yml" ]
+    grep -q '^  http_2xx:' "$sandbox/blackbox.yml"
+}
+
+@test "the alloy copier refuses to install a config the engine rejects" {
+    # The syntax gate: alloy attributes are position-sensitive and a
+    # misplaced one only explodes at run time (crash-loop behind a green
+    # deploy). Prove the copier refuses a config the pinned image won't
+    # validate, and installs nothing. Needs the alloy image; skipped
+    # where it is absent (the copier's gate is likewise image-gated).
+    if ! podman image exists localhost/franksdank/alloy:latest; then
+        skip "franksdank/alloy image not present on this box"
+    fi
+    local sandbox="$BATS_TEST_TMPDIR/opt" cfg="$BATS_TEST_TMPDIR/broken.alloy"
+    sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
+    printf 'bogus_attr = "x"\n' >> "$cfg"
+    ! (
+        cd "$REPO_ROOT"
+        eval "$(sed -n '/^copy_alloy_config()/,/^}/p' scripts/deploy-remote.sh)"
+        run_privileged() { "$@"; }
+        cp() { if [ "$1" = "$cfg" ]; then return 1; fi; command cp "$@"; }
+        export TARGET=dankclaw ALLOY_CONF_ROOT="$sandbox"
+        copy_alloy_config
+    )
+    [ ! -f "$sandbox/config/config.alloy" ]
+}
+
+@test "the installed alloy config is engine-valid" {
+    # Runs the pinned image's own validator over the shipped config. The
+    # 2026-10-21 defect (`on_error` on the labels stage, which alloy 1.21
+    # does not define) passed every textual seam and only failed at boot.
+    if ! podman image exists localhost/franksdank/alloy:latest; then
+        skip "franksdank/alloy image not present on this box"
+    fi
+    local cfg="$BATS_TEST_TMPDIR/rendered.alloy"
+    sed 's/GRAFANA_HOST_LABEL/dankclaw/g' shared/alloy/config.alloy > "$cfg"
+    podman run --rm -v "$cfg:/c.alloy:ro" --entrypoint /usr/bin/alloy \
+        localhost/franksdank/alloy:latest validate /c.alloy
+}
@@ -15,8 +15,9 @@ QDIR="$REPO_ROOT/dankclaw/open-webui/quadlet"
 }

@test "open-webui uses the CPU image, not :cuda (dankclaw is CPU-only)" {
-    grep -qE '^Image=ghcr\.io/open-webui/open-webui:main$' "$QDIR/open-webui.container" \
-        || { echo "open-webui.container must use the :main (CPU) image"; return 1; }
+    # 2026-10-21 fleet law: tag AND digest (tests/quadlet_image_pins.bats)
+    grep -qE '^Image=ghcr\.io/open-webui/open-webui:main@sha256:[0-9a-f]{64}$' "$QDIR/open-webui.container" \
+        || { echo "open-webui.container must use the :main (CPU) image, digest-pinned"; return 1; }
    if grep -qE '^Image=.*:cuda' "$QDIR/open-webui.container"; then
        echo "open-webui.container must not use the :cuda image on a CPU-only host"
        return 1
@@ -0,0 +1,47 @@
+#!/usr/bin/env bats
+# Seam: fleet-wide quadlet laws learned the hard way.
+#   1. Every registry image is digest-pinned. A rolling tag (":latest",
+#      ":9-alpine", ":nightly") lets a rebuild move under a running
+#      service at the next restart/reboot with nobody approving it, and it
+#      makes a CVE audit ungradeable (you cannot score a tag). Locally
+#      built images (localhost/*, franksdank/*) are exempt: their build
+#      IS the pin, and they never pull.
+#   2. No unit carries a key the podman 5.7.0 quadlet generator rejects.
+#      The generator aborts the WHOLE batch on an unsupported key, so one
+#      bad file silently un-generates every other unit in that manager -
+#      live 2026-10-21: HostAddr= in prometheus.container orphaned
+#      prometheus.service while its container kept running, and the next
+#      deploy died on "Unit prometheus.service not found".
+
+load helpers
+
+@test "every registry Image= is digest-pinned across all three hosts" {
+    bad=""
+    while IFS= read -r unit; do
+        while IFS= read -r img; do
+            case "$img" in
+                localhost/*|franksdank/*) continue ;;
+            esac
+            case "$img" in
+                *@sha256:*) ;;
+                *) bad="$bad$unit: $img"$'\n' ;;
+            esac
+        done < <(sed -n 's/^Image=//p' "$unit")
+    done < <(find "$REPO_ROOT"/dankserver "$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro \
+                 -path '*/quadlet/*.container' -not -path '*.AGENTTRASH*')
+    [ -z "$bad" ] || { printf 'unpinned images:\n%s' "$bad" >&2; return 1; }
+}
+
+@test "no unit uses a key the 5.7.0 quadlet generator rejects (batch killer)" {
+    # HostAddr rejected by podman 5.7.0 (live 2026-10-21); Device= and
+    # Command= are the older v1 findings, guarded host-locally too.
+    # Keep this list current with every generator finding.
+    bad=""
+    while IFS= read -r unit; do
+        for key in HostAddr Device Command; do
+            grep -q "^${key}=" "$unit" && bad="$bad$unit: ${key}="$'\n'
+        done
+    done < <(find "$REPO_ROOT"/dankserver "$REPO_ROOT"/dankclaw "$REPO_ROOT"/frankbook-pro \
+                 -path '*/quadlet/*.container' -not -path '*.AGENTTRASH*')
+    [ -z "$bad" ] || { printf 'generator-rejected keys:\n%s' "$bad" >&2; return 1; }
+}
@@ -20,8 +20,9 @@ SEERR_UNIT="$REPO_ROOT/dankclaw/seerr/quadlet/seerr.container"
    grep -q '^Restart=always$' "$SEERR_UNIT"
 }

-@test "seerr image is pinned to an exact version tag" {
-    grep -Eq '^Image=ghcr\.io/seerr-team/seerr:v[0-9]+\.[0-9]+\.[0-9]+$' "$SEERR_UNIT"
+@test "seerr image is pinned to an exact version tag and digest" {
+    # 2026-10-21 fleet law: tag AND digest (tests/quadlet_image_pins.bats)
+    grep -Eq '^Image=ghcr\.io/seerr-team/seerr:v[0-9]+\.[0-9]+\.[0-9]+@sha256:[0-9a-f]{64}$' "$SEERR_UNIT"
 }

@test "seerr publishes 5055 on loopback only (guard chain is the user path)" {
Author	SHA1	Message	Date
dank	0ecfe830f7	Merge pull request 'fix(deploy): relay firewall DROP must be appended, not -I-inserted above its ACCEPTs' (#75 ) from fix/signal-relay-fw-order into main	2026-10-21 16:59:16 -07:00
Frank	ce370a5b28	fix(deploy): relay firewall DROP must be appended, not -I-inserted above its ACCEPTs iptables -I prepends, so the install block's final -I of the blanket DROP landed it at position 1, above all three ACCEPTs: every packet to 18093/tcp died there (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT). The host's own health probe timed out beside a LISTENing socket and the whole dankclaw drain aborted at the alertmanager gate twice. Install now rebuilds the tail: delete any blanket DROP wherever it sits, keep the ACCEPTs at the head, append the DROP last - converges on hosts poisoned by the old order. New bats case runs install_signal_relay for real against an iptables sandbox seeded with the poisoned order (red on the old code, green on this one).	2026-10-21 16:52:40 -07:00
Frank	bfe4d185bd	docs(vault): OpenWebUI picker opened to all three models for every account (2026-10-21) Per Frank, frank-archon is public alongside dragoon/zealot: three-leg DB trio (conn enable, model row ctx 200000, access_grant principal='*'), verified as role=user. Token-mint traps recorded (cwd, PID-1 secret).	2026-10-21 16:48:32 -07:00
dank	94a9f3aa31	Merge pull request 'feat(security): digest-pin every registry image; drop batch-killing HostAddr' (#74 ) from security/quadlet-digest-pins into main	2026-10-21 16:32:44 -07:00
Frank	d2810ab610	feat(security): digest-pin every registry image on all three hosts (fleet law) Closing the command-center security queue #70/#71/#72/#73/#74: rolling tags let a rebuild move under a running service with nobody approving it, and make a CVE audit ungradeable. 36 quadlet units across dankclaw, dankserver and frankbook-pro now carry tag@sha256; localhost/* builds stay exempt (their build is the pin). The pihole and copyparty pins pull the fixed builds; valkey 9.1.2 (already running) and redis 7.4.11 are pinned in place, vendor fixes TBD. New gate tests/quadlet_image_pins.bats: no unpinned registry image, no generator-rejected key anywhere. Three seams that pinned the old exact-tag law (frankbook readarr, open_webui, seerr) now require tag AND digest - strictly stronger. Quadlet Lifecycle docs: batch-abort law, entrypoint flags-only law, pin law.	2026-10-21 16:32:36 -07:00
Frank	7a8cc83385	fix(quadlet): drop HostAddr from prometheus.unit - podman 5.7.0 rejects the key and the quadlet generator then aborts the ENTIRE user batch, silently orphaning every other unit (prometheus.service vanished while the container ran; deploy died 'Unit prometheus.service not found', live 2026-10-21). host.containers.internal resolves natively; the key was never needed.	2026-10-21 16:32:36 -07:00
dank	c4623f1629	Merge pull request 'fix(blackbox): flags-only Exec; bind loopback+LAN; exec-form HealthCmd' (#73 ) from fix/blackbox-entrypoint-bind into main	2026-10-21 16:25:32 -07:00
Frank	065a6911f9	fix(blackbox): Exec carries flags only (image has ENTRYPOINT) and the exporter binds loopback+LAN Three stacked lessons, each proved live 2026-10-21: - the image's ENTRYPOINT is /bin/blackbox_exporter, so Exec= repeating the binary made it a positional arg: 'unexpected /bin/blackbox_exporter'. - the podman gateway 169.254.1.2 is NOT bindable from the host netns - it exists only inside each rootless netns: 'bind: cannot assign requested address'. The scrape path does not need it bound: a bridge container dialing the gateway lands on the host LOOPBACK (proved 200 from the prometheus container), so the exporter binds 127.0.0.1 + LAN. - HealthCmd shell form ('/bin/sh -c "..."') is passed as one quoted token and quadlet folds the following --health-interval/--health-retries (and the image) into the command string; the plain exec form the sibling units use is correct. Prometheus.md bind law corrected (gateway is a forward-to-loopback, not a bind address); deploy-remote gate stays loopback, which is now honest.	2026-10-21 16:04:53 -07:00
dank	745d66d60d	Merge pull request 'fix(blackbox): --web.listen-address (repeatable), not --listen-address' (#72 ) from fix/blackbox-web-listen-flag into main	2026-10-21 15:58:36 -07:00
Frank	4009f118c1	fix(blackbox): the flag is --web.listen-address, repeatable, not --listen-address Third boot failure: the exporter exited 100 ms in with 'unknown long flag --listen-address' (proved against the pinned image, plus --help). The bind semantics were right; the flag name was invented. --web.listen-address is repeatable - one per address - so the dual gateway+LAN bind stands. Proven again pre-deploy with the real argv: the run starts and dies only on the deliberate /dev/null placeholder config.	2026-10-21 15:55:21 -07:00
dank	7de343c171	Merge pull request 'fix(blackbox): complete the unit - Image, Network=host, honest bind topology' (#71 ) from fix/blackbox-unit-generation into main	2026-10-21 15:52:30 -07:00
Frank	139a3cd4fa	fix(blackbox): unit had no Image and no Network=host - quadlet refused to generate it The deploy died at 'Unit blackbox-exporter.service not found': a .container without Image= is not generated at all, and the header argued for Network=host it never declared. Completes the unit (digest pinned prom/blackbox-exporter, Network=host) and fixes the bind topology the first cut got wrong: - Prometheus is bridge-networked and the netavark hairpin gap bars it from its own host's LAN IP, so a LAN-only prober bind is unscrapable and a loopback-only bind was claimed to ride the gateway on faith. The prober now binds 9115 on the podman gateway (scrape path) AND the LAN (healthcheck/diagnostics); the listen_address line leaves blackbox.yml so config and Exec cannot disagree. - comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188: bridge container names do not resolve in the host netns; the published 0.0.0.0:8188 answers on loopback there. - New seam: every observability unit must declare Image= (red on the shipped file by name); copier test pinned module presence instead of the removed listen line. Docs updated to the real topology.	2026-10-21 15:41:37 -07:00
dank	5ffbf19fb2	Merge pull request 'fix(deploy): blackbox-exporter was gated out of its own deploy path' (#70 ) from fix/blackbox-service-gates into main	2026-10-21 15:34:49 -07:00
Frank	1fd98cab3f	fix(deploy): blackbox-exporter was gated out of its own deploy path PR #69 wired the copier, planner entry and health gate but missed the three dispatch gates: the client valid_service, the remote service_is_allowed, and deploy_service's case list. The client gate rejected --service blackbox-exporter outright; even past it the dispatcher's catch-all hard-fails on dankclaw (no compose since 2026-09-16). New seam pins the class: every dankclaw service the remote gate allows must carry a deploy_service case - proven red by planting a gated, undispatched name.	2026-10-21 14:59:47 -07:00
dank	d61c19d41f	Merge pull request 'Observability: kill every structural error (blackbox, labels, gates)' (#69 ) from fix/observability-errors-gone into main	2026-10-21 14:53:16 -07:00
Frank	e6fcf217b4	docs(comfyui): the GTT aperture is not free RAM beside frank-dragoon Measured 2026-10-21 with frank-dragoon live: MemAvailable 15.3 GiB, iGPU busy 1.75 GiB. The Qwen-Image weight sets (13.3-14.2 GB) do not fit that budget, and the page's 122 GB /system_stats figure is the aperture, not headroom. Adds the fit table and the stop-mask-render-verify procedure (the 3090 bench law, applied to the APU); ComfyUI.md gotcha links it.	2026-10-21 13:35:50 -07:00
Frank	cb86ad8fd7	docs(observability): post-blackbox state, copier law, label-promotion why Prometheus/Observability/Provisioning pages: every job up after the blackbox swap, the copier list learns copy_blackbox_config and the alloy validate gate, the config table gains alloy + blackbox rows, and the logs bullet states the stream-label law instead of the retired 'awaiting deploy' note.	2026-10-21 13:33:03 -07:00
Frank	078b550bd8	test(observability): copier sandboxes, probe-target seam exception, cop gates - caddyfile seam learns the documented exception: URL-shaped targets are legal only inside a prober job (state is per-job; the first cut leaked it and passed a planted URL). - observability seams: blackbox unit + planner entries, both copiers executed for real in sandboxes, alloy engine-validation refusal (image-gated), dashboard panel invariants. - AGENTS.md: structural target downs retired.	2026-10-21 13:33:03 -07:00
Frank	765a8cb7de	feat(deploy): blackbox copier, alloy engine-validation gate, planner wiring copy_alloy_config now validates the rendered config with the very alloy image that will run it, before installing: a misplaced attribute crash-loops the unit behind a green deploy, and no textual seam can see it. copy_blackbox_config ships the prober config; the planner maps the new unit and shared/blackbox/** to the blackbox-exporter service, ordered before prometheus.	2026-10-21 13:33:03 -07:00
Frank	999c4fe440	fix(alloy): promote CONTAINER_NAME to an indexed Loki stream label Loki metrics queries group only by indexed stream labels, so the journal-as-JSON stream made every per-container view structurally empty (measured 6.5k lines/5min flowing while the board read zero). A loki.process relabel stage now extracts CONTAINER_NAME into a 'container' label. alloy 1.21's json stage has no on_error attribute (validated against the pinned image) — format_as_json guarantees the line shape, so no error branch exists.	2026-10-21 13:33:03 -07:00
Frank	2af2e6ac56	fix(grafana): observability and logs-overview boards read real data observability.json: stale 'structural downs' panel description fixed, target census matches the post-blackbox job set. logs-overview.json: panels grouped by the promoted container label instead of the absent container_name, so the board stops reading empty beside live traffic.	2026-10-21 13:32:58 -07:00
Frank	fbe54ca1ad	feat(gitea): enable /metrics for Prometheus scraping Gitea shipped metrics disabled, so its scrape target was a permanent structural down beside a healthy service.	2026-10-21 13:32:58 -07:00
Frank	f073b91033	feat(observability): blackbox exporter replaces the HTML-probe jobs Prometheus 3 refuses HTML scrape bodies, so the comfyui/services probe jobs were structurally dead targets reading as outages. HTTP liveness now rides a blackbox exporter (shared/blackbox/blackbox.yml, loopback bind, http_2xx module); prometheus.yml probes SearXNG, OpenWebUI and ComfyUI via __param_target relabeling. The unit is a boot-persistent, self-healing root quadlet.	2026-10-21 13:32:58 -07:00
dank	425994515a	Merge pull request 'fix(deploy): grafana copier provisions EVERY dashboard json' (#68 ) from fix/grafana-copier-glob into main	2026-10-21 09:49:15 -07:00
Frank	c41384e03d	fix(deploy): grafana copier provisions EVERY dashboard json, not a hand-listed subset copy_grafana_provisioning named llm-fleet.json alone, so a new board in shared/grafana/provisioning/dashboards/ silently never reached /opt/grafana/provisioning. Measured 2026-10-21: the containers board was merged, seam-tested, and documented as working since 2026-10-20, yet was NEVER provisioned — its panel fixes were undeployable; the other four boards existed on disk only because someone hand-copied them once (and host-system.json there had gone stale against the repo). The copier now globs the directory, and GRAFANA_PROV_ROOT lets the new bats case run the copier for real into a sandbox (red against the old hand-listed body, green against the glob). Vault: the copier law + the containers incident on Observability Provisioning, the corrected dashboards claim on Observability. Note: a mktemp sandbox + trap RETURN races bats' trap machinery (the dir vanishes mid-function); BATS_TEST_TMPDIR is the safe idiom.	2026-10-21 09:40:57 -07:00
dank	127621bdd7	Merge pull request 'docs(vault)+fix(grafana): audit tranche 1 and dashboard data-flow fixes' (#67 ) from fix/dragoon-context-200k into main	2026-10-21 09:28:43 -07:00
dank	ccb805cf0b	Merge pull request 'fix(frank-dragoon): mount the distro XRT at the third prefix the NPU child searches' (#66 ) from fix/dragoon-context-200k into main	2026-10-21 02:25:42 -07:00
dank	9eb53ef937	Merge pull request 'feat(frank-dragoon): 200,000-token harness window via the native 262,144 ceiling' (#65 ) from fix/dragoon-context-200k into main	2026-10-21 01:40:40 -07:00