fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway

Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:

- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
  serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
  but never accept()ed, loopback deploy gate passed, every scrape that
  reached the other bind timed out on the SYN backlog. Now one thread per
  extra bind. Regression test spins --serve on two loopback binds and
  requires BOTH to answer /metrics.

- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
  Prometheus container: a bridge container cannot dial its own host's LAN
  address at all (hairpin refuses published ports too); the podman gateway
  answers. Target moved to host.containers.internal:8080; the seam that
  pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
  gateway plus a negative pin against the own-host LAN dial. Observability
  page records the measured reachability law replacing the inferred one.
This commit is contained in:
Frank
2026-10-21 00:34:17 -07:00
parent b62a05837f
commit 650b6e0281
5 changed files with 68 additions and 15 deletions
@@ -67,10 +67,18 @@ Alertmanager has no vhost at all. node-exporter is the exception that must live
in the host network namespace, and it binds exactly two addresses rather than
the wildcard — see [[Node-Exporter]].
The relay's bind is the one address a bridge container can reach on the host:
`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
and the podman bridge subnet `10.89.0.0/16` before the DROP
(`scripts/deploy-remote.sh:255-262`).
**Same-host reachability from a bridge container is measured, not inferred**
(2026-10-21, from inside the Prometheus container): `host.containers.internal`
(the podman gateway) reaches host services whether they bind host-net or are
published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
every same-host Prometheus target uses the gateway and only cross-host targets
(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
iptables INPUT admitting loopback, the host itself and the podman bridge subnet
`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
point a scrape target at a host LAN IP from its own host on the strength of
one working example; dial the gateway and probe.
## Deployment