fix(blackbox): unit had no Image and no Network=host - quadlet refused to generate it
The deploy died at 'Unit blackbox-exporter.service not found': a .container without Image= is not generated at all, and the header argued for Network=host it never declared. Completes the unit (digest pinned prom/blackbox-exporter, Network=host) and fixes the bind topology the first cut got wrong: - Prometheus is bridge-networked and the netavark hairpin gap bars it from its own host's LAN IP, so a LAN-only prober bind is unscrapable and a loopback-only bind was claimed to ride the gateway on faith. The prober now binds 9115 on the podman gateway (scrape path) AND the LAN (healthcheck/diagnostics); the listen_address line leaves blackbox.yml so config and Exec cannot disagree. - comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188: bridge container names do not resolve in the host netns; the published 0.0.0.0:8188 answers on loopback there. - New seam: every observability unit must declare Image= (red on the shipped file by name); copier test pinned module presence instead of the removed listen line. Docs updated to the real topology.
This commit is contained in:
1 parent
5ffbf19fb2
commit
139a3cd4fa
8 files changed
+59
-35
No files matched your search
@@ -1,29 +1,38 @@
|
||||
# Quadlet unit: blackbox exporter (rootless, boot-persistent).
|
||||
# Quadlet unit: blackbox exporter (rootless, host network, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw:
|
||||
# ~/.config/containers/systemd/blackbox-exporter.container
|
||||
#
|
||||
# WHY Network=host. The probes it runs are (a) the host's own LAN IP
|
||||
# (192.168.0.143:<port>) and (b) container names on dankclaw_default (comfyui).
|
||||
# A bridge container cannot reach the host's LAN IP at an unpublished port
|
||||
# (netavark DNATs published ports only — see Container Host Reachability in the
|
||||
# vault); a host-netns process reaches both. Prometheus (bridge) reaches THIS
|
||||
# container the same way it reaches the signal relay: the quadlet-published
|
||||
# host.containers.internal address (169.254.1.2), mapped via HostAddr= on the
|
||||
# prometheus unit.
|
||||
# (192.168.0.143:8081 SearXNG, :3005 OpenWebUI — host-process/portmap binds a
|
||||
# bridge container cannot reach; netavark DNATs published ports only, see
|
||||
# Container Host Reachability in the vault) and (b) ComfyUI's published port
|
||||
# via 127.0.0.1 (rootless --network=host shares the real host netns, so the
|
||||
# 0.0.0.0:8188 publish answers on loopback). A bridge-networked prober could
|
||||
# reach neither class reliably.
|
||||
#
|
||||
# The 9115 metrics port publishes on loopback only — it is scraped by the
|
||||
# prometheus container through the gateway, and never opened on the LAN.
|
||||
# WHY the listener binds TWO addresses. The netavark hairpin gap (measured
|
||||
# 2026-10-21, cadvisor job comment): a BRIDGE CONTAINER CANNOT DIAL ITS OWN
|
||||
# HOST'S LAN IP at all — so a LAN-only bind is unscrapable by Prometheus,
|
||||
# whatever the signal-relay law suggests for services containers dial as
|
||||
# clients. The podman gateway (169.254.1.2 = host.containers.internal) is the
|
||||
# address bridge containers reach same-host targets at, so the exporter binds
|
||||
# it explicitly for the scrape rather than relying on loopback-forwarding
|
||||
# semantics, and keeps the LAN bind for its own healthcheck and diagnostics.
|
||||
# No public DNS name, no guard chain: internal net only, per the exposure
|
||||
# policy.
|
||||
|
||||
[Unit]
|
||||
Description=Blackbox exporter (HTTP probe prober)
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
Image=docker.io/prom/blackbox-exporter@sha256:ea1c7aa3ef3f307cd03e85f34abb604478ccb90b567fc690da0cd9776da76b13
|
||||
ContainerName=blackbox-exporter
|
||||
Exec=/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/config.yml
|
||||
Network=host
|
||||
Exec=/bin/blackbox_exporter --config.file=/etc/blackbox_exporter/config.yml --listen-address=192.168.0.143:9115,169.254.1.2:9115
|
||||
Volume=/opt/blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
|
||||
HealthCmd=/bin/sh -c "wget -q -O /dev/null http://127.0.0.1:9115/-/healthy || exit 1"
|
||||
HealthCmd=/bin/sh -c "wget -q -O /dev/null http://169.254.1.2:9115/-/healthy || exit 1"
|
||||
HealthInterval=30s
|
||||
HealthRetries=3
|
||||
|
||||
|
||||
@@ -45,10 +45,11 @@ Wants=network-online.target
|
||||
ContainerName=prometheus
|
||||
Image=docker.io/prom/prometheus:latest
|
||||
Network=dankclaw_default
|
||||
# Name the podman gateway so scrape jobs can reach host-netns processes (the
|
||||
# blackbox exporter listens on the host's loopback; a bridge container reaches
|
||||
# it at this address, never at 127.0.0.1). Same mechanism the hermes Caddy
|
||||
# upstream learned the hard way (Container Host Reachability in the vault).
|
||||
# Name the podman gateway so scrape jobs can reach same-host targets (the
|
||||
# netavark hairpin gap bars this container from dialing its own host's LAN
|
||||
# IP; the cadvisor/gpu jobs and the blackbox scrape all ride this name).
|
||||
# Same mechanism the hermes Caddy upstream learned the hard way (Container
|
||||
# Host Reachability in the vault).
|
||||
HostAddr=host.containers.internal=169.254.1.2
|
||||
User=0:0
|
||||
Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
|
||||
|
||||
Reference in new issue
Block a user