From 065a6911f9f5dec7a15fd91d5bd919b2bdbf427f Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 21 Oct 2026 16:04:53 -0700 Subject: [PATCH] fix(blackbox): Exec carries flags only (image has ENTRYPOINT) and the exporter binds loopback+LAN Three stacked lessons, each proved live 2026-10-21: - the image's ENTRYPOINT is /bin/blackbox_exporter, so Exec= repeating the binary made it a positional arg: 'unexpected /bin/blackbox_exporter'. - the podman gateway 169.254.1.2 is NOT bindable from the host netns - it exists only inside each rootless netns: 'bind: cannot assign requested address'. The scrape path does not need it bound: a bridge container dialing the gateway lands on the host LOOPBACK (proved 200 from the prometheus container), so the exporter binds 127.0.0.1 + LAN. - HealthCmd shell form ('/bin/sh -c "..."') is passed as one quoted token and quadlet folds the following --health-interval/--health-retries (and the image) into the command string; the plain exec form the sibling units use is correct. Prometheus.md bind law corrected (gateway is a forward-to-loopback, not a bind address); deploy-remote gate stays loopback, which is now honest. --- .../quadlet/blackbox-exporter.container | 38 ++++++++++++------- scripts/deploy-remote.sh | 2 + .../Infrastructure/Prometheus.md | 7 ++-- 3 files changed, 30 insertions(+), 17 deletions(-) diff --git a/dankclaw/observability/quadlet/blackbox-exporter.container b/dankclaw/observability/quadlet/blackbox-exporter.container index 5d5b9f6..c590d96 100644 --- a/dankclaw/observability/quadlet/blackbox-exporter.container +++ b/dankclaw/observability/quadlet/blackbox-exporter.container @@ -11,18 +11,19 @@ # 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could # reach neither class reliably. # -# WHY the listener binds TWO addresses. The netavark hairpin gap (measured -# 2026-10-21, cadvisor job comment): a BRIDGE CONTAINER CANNOT DIAL ITS OWN -# HOST'S LAN IP at all — so a LAN-only bind is unscrapable by Prometheus, -# whatever the signal-relay law suggests for services containers dial as -# clients. The podman gateway (169.254.1.2 = host.containers.internal) is the -# address bridge containers reach same-host targets at, so the exporter binds -# it explicitly for the scrape rather than relying on loopback-forwarding -# semantics, and keeps the LAN bind for diagnostics. No public DNS name, no -# guard chain: internal net only, per the exposure policy. The listen flag is -# --web.listen-address and it is REPEATABLE, one flag per bind; plain -# --listen-address does not exist and the exporter exits in 100 ms with -# "unknown long flag" (live 2026-10-21). +# WHY the listener binds 127.0.0.1 + LAN. A bridge container CANNOT dial +# its own host's LAN IP (netavark hairpin gap, measured 2026-10-21), but a +# bridge container dialing the podman gateway 169.254.1.2 lands on the host +# LOOPBACK — that is the scrape path, and it needs this exporter to be +# LISTENING ON LOOPBACK (proved: 200 from the prometheus container to +# http://169.254.1.2:9115/-/healthy with a loopback bind). The gateway +# address itself is NOT bindable from the host netns — it exists only +# inside each rootless netns; binding it dies "bind: cannot assign +# requested address" (live 2026-10-21). The LAN bind stays for diagnostics. +# No public DNS name, no guard chain: internal net only, per the exposure +# policy. The listen flag is --web.listen-address and it is REPEATABLE, one +# flag per bind; plain --listen-address does not exist and the exporter +# exits in 100 ms with "unknown long flag" (live 2026-10-21). [Unit] Description=Blackbox exporter (HTTP probe prober) @@ -32,9 +33,18 @@ After=network-online.target Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13 ContainerName=blackbox-exporter Network=host -Exec=/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/config.yml --web.listen-address=192.168.0.143:9115 --web.listen-address=169.254.1.2:9115 +# The image's ENTRYPOINT is /bin/blackbox_exporter itself (proved live +# 2026-10-21: `podman run IMAGE sh` dies "unexpected sh"), so Exec= carries +# FLAGS ONLY — repeating the binary makes it a positional arg and kingpin +# exits "unexpected /bin/blackbox_exporter". +Exec=--config.file=/etc/blackbox_exporter/config.yml --web.listen-address=127.0.0.1:9115 --web.listen-address=192.168.0.143:9115 Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro -HealthCmd=/bin/sh -c "wget -q -O /dev/null http://169.254.1.2:9115/-/healthy || exit 1" +# Plain exec form like every sibling unit. Shell form (`/bin/sh -c "…"`) +# is passed as one quoted token and quadlet then folds the following +# --health-interval/--health-retries (and the image name) into the command +# string — unit died twice that way, live 2026-10-21. wget only; this +# busybox variant has no curl, and host-network wget answers on loopback. +HealthCmd=wget -q -O /dev/null http://127.0.0.1:9115/-/healthy HealthInterval=30s HealthRetries=3 diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 24ca136..777ec0a 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -617,6 +617,8 @@ deploy_observability() { loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;; node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;; cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;; + # Loopback: the exporter binds 127.0.0.1 + LAN (the gateway address + # is not bindable from the host netns — see the unit header). blackbox-exporter) wait_for_health blackbox-exporter http://127.0.0.1:9115/-/healthy 60 ;; esac } diff --git a/serverAppConfigs-docs/Infrastructure/Prometheus.md b/serverAppConfigs-docs/Infrastructure/Prometheus.md index dc47a11..4175463 100644 --- a/serverAppConfigs-docs/Infrastructure/Prometheus.md +++ b/serverAppConfigs-docs/Infrastructure/Prometheus.md @@ -54,9 +54,10 @@ Scraping a web root is not a health check: an HTML page fails the scrape protocol (`unsupported Content-Type`), which is what kept those targets red beside healthy services. The classification rule and the honesty law live in [[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host -netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on both the LAN -and the podman gateway, and Prometheus scrapes the gateway address — the -netavark hairpin gap bars this container from its own host's LAN IP. +netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on loopback and +the LAN, and Prometheus scrapes it at `host.containers.internal` — traffic to +that gateway lands on the host's loopback listener. The gateway address is +not itself bindable from the host netns. Plain targets stay `host:port`, never URLs — the probe job's URL targets are the documented exception, relabeled onto the prober.