feat(observability): blackbox exporter replaces the HTML-probe jobs

Prometheus 3 refuses HTML scrape bodies, so the comfyui/services probe
jobs were structurally dead targets reading as outages. HTTP liveness
now rides a blackbox exporter (shared/blackbox/blackbox.yml, loopback
bind, http_2xx module); prometheus.yml probes SearXNG, OpenWebUI and
ComfyUI via __param_target relabeling. The unit is a boot-persistent,
self-healing root quadlet.

dankclaw/observability/quadlet/blackbox-exporter.container

@@ -0,0 +1,35 @@
+# Quadlet unit: blackbox exporter (rootless, boot-persistent).
+#
+# Install path on dankclaw:
+#   ~/.config/containers/systemd/blackbox-exporter.container
+#
+# WHY Network=host. The probes it runs are (a) the host's own LAN IP
+# (192.168.0.143:<port>) and (b) container names on dankclaw_default (comfyui).
+# A bridge container cannot reach the host's LAN IP at an unpublished port
+# (netavark DNATs published ports only — see Container Host Reachability in the
+# vault); a host-netns process reaches both. Prometheus (bridge) reaches THIS
+# container the same way it reaches the signal relay: the quadlet-published
+# host.containers.internal address (169.254.1.2), mapped via HostAddr= on the
+# prometheus unit.
+#
+# The 9115 metrics port publishes on loopback only — it is scraped by the
+# prometheus container through the gateway, and never opened on the LAN.
+
+[Unit]
+Description=Blackbox exporter (HTTP probe prober)
+After=network-online.target
+
+[Container]
+ContainerName=blackbox-exporter
+Exec=/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/config.yml
+Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
+HealthCmd=/bin/sh -c "wget -q -O /dev/null http://127.0.0.1:9115/-/healthy || exit 1"
+HealthInterval=30s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/prometheus.container

@@ -45,6 +45,11 @@ Wants=network-online.target
 ContainerName=prometheus
 Image=docker.io/prom/prometheus:latest
 Network=dankclaw_default
+# Name the podman gateway so scrape jobs can reach host-netns processes (the
+# blackbox exporter listens on the host's loopback; a bridge container reaches
+# it at this address, never at 127.0.0.1). Same mechanism the hermes Caddy
+# upstream learned the hard way (Container Host Reachability in the vault).
+HostAddr=host.containers.internal=169.254.1.2
 User=0:0
 Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
 PublishPort=127.0.0.1:9090:9090