fix(blackbox): Exec carries flags only (image has ENTRYPOINT) and the exporter binds loopback+LAN

Three stacked lessons, each proved live 2026-10-21:
- the image's ENTRYPOINT is /bin/blackbox_exporter, so Exec= repeating the
  binary made it a positional arg: 'unexpected /bin/blackbox_exporter'.
- the podman gateway 169.254.1.2 is NOT bindable from the host netns - it
  exists only inside each rootless netns: 'bind: cannot assign requested
  address'. The scrape path does not need it bound: a bridge container
  dialing the gateway lands on the host LOOPBACK (proved 200 from the
  prometheus container), so the exporter binds 127.0.0.1 + LAN.
- HealthCmd shell form ('/bin/sh -c "..."') is passed as one quoted token
  and quadlet folds the following --health-interval/--health-retries (and
  the image) into the command string; the plain exec form the sibling units
  use is correct.

Prometheus.md bind law corrected (gateway is a forward-to-loopback, not a
bind address); deploy-remote gate stays loopback, which is now honest.

dankclaw/observability/quadlet/blackbox-exporter.container

@@ -11,16 +11,19 @@
 # 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
 # reach neither class reliably.
 #
-# WHY the listener binds TWO addresses. The netavark hairpin gap (measured
+# WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial
-# 2026-10-21, cadvisor job comment): a BRIDGE CONTAINER CANNOT DIAL ITS OWN
+# its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a
-# HOST'S LAN IP at all — so a LAN-only bind is unscrapable by Prometheus,
+# bridge container dialing the podman gateway 169.254.1.2 lands on the host
-# whatever the signal-relay law suggests for services containers dial as
+# LOOPBACK — that is the scrape path, and it needs this exporter to be
-# clients. The podman gateway (169.254.1.2 = host.containers.internal) is the
+# LISTENING ON LOOPBACK (proved: 200 from the prometheus container to
-# address bridge containers reach same-host targets at, so the exporter binds
+# http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway
-# it explicitly for the scrape rather than relying on loopback-forwarding
+# address itself is NOT bindable from the host netns — it exists only
-# semantics, and keeps the LAN bind for its own healthcheck and diagnostics.
+# inside each rootless netns; binding it dies "bind: cannot assign
+# requested address" (live 2026-10-21). The LAN bind stays for diagnostics.
 # No public DNS name, no guard chain: internal net only, per the exposure
-# policy.
+# policy. The listen flag is --web.listen-address and it is REPEATABLE, one
+# flag per bind; plain --listen-address does not exist and the exporter
+# exits in 100 ms with "unknown long flag" (live 2026-10-21).
 
 [Unit]
 Description=Blackbox exporter (HTTP probe prober)
@@ -30,9 +33,18 @@ After=network-online.target
 Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
 ContainerName=blackbox-exporter
 Network=host
-Exec=/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/config.yml --listen-address=192.168.0.143:9115,169.254.1.2:9115
+# The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live
+# 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries
+# FLAGS ONLY — repeating the binary makes it a positional arg and kingpin
+# exits "unexpected /bin/blackbox_exporter".
+Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115
 Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
-HealthCmd=/bin/sh -c "wget -q -O /dev/null http://169.254.1.2:9115/-/healthy || exit 1"
+# Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`)
+# is passed as one quoted token and quadlet then folds the following
+# --health-interval/--health-retries (and the image name) into the command
+# string — unit died twice that way, live 2026-10-21. wget only; this
+# busybox variant has no curl, and host-network wget answers on loopback.
+HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy
 HealthInterval=30s
 HealthRetries=3
 

scripts/deploy-remote.sh

@@ -617,6 +617,8 @@ deploy_observability() {
         loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
         node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
         cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
+        # Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address
+        # is not bindable from the host netns — see the unit header).
         blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;;
     esac
 }

serverAppConfigs-docs/Infrastructure/Prometheus.md

@@ -54,9 +54,10 @@ Scraping a web root is not a health check: an HTML page fails the scrape
 protocol (`unsupported Content-Type`), which is what kept those targets red
 beside healthy services. The classification rule and the honesty law live in
 [[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
-netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on both the LAN
+netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on loopback and
-and the podman gateway, and Prometheus scrapes the gateway address — the
+the LAN, and Prometheus scrapes it at `host.containers.internal` — traffic to
-netavark hairpin gap bars this container from its own host's LAN IP.
+that gateway lands on the host's loopback listener. The gateway address is
+not itself bindable from the host netns.
 Plain targets stay `host:port`, never URLs — the probe job's URL targets are
 the documented exception, relabeled onto the prober.