fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway

Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:

- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
  serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
  but never accept()ed, loopback deploy gate passed, every scrape that
  reached the other bind timed out on the SYN backlog. Now one thread per
  extra bind. Regression test spins --serve on two loopback binds and
  requires BOTH to answer /metrics.

- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
  Prometheus container: a bridge container cannot dial its own host's LAN
  address at all (hairpin refuses published ports too); the podman gateway
  answers. Target moved to host.containers.internal:8080; the seam that
  pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
  gateway plus a negative pin against the own-host LAN dial. Observability
  page records the measured reachability law replacing the inferred one.
This commit is contained in:
Frank committed 2026-10-21 00:34:17 -07:00
1 parent b62a05837f
commit 650b6e0281
5 files changed
+68 -15

No files matched your search

+30
View File
@@ -195,3 +195,33 @@ for want in ('gpu_utilization_percent', 'gpu_memory_used_bytes',
print('gpu dashboard panels ok')
"
}
@test "--serve answers on EVERY declared bind, not just the first" {
# The deployed unit serves loopback + the host LAN address, and Prometheus
# reaches the host through the gateway — which dials the LAN-class bind.
# A server that only runs serve_forever() on the first socket leaves the
# second one listen()ing-but-never-accepting: the deploy health gate on
# loopback passes, every scrape on the other bind dies on the SYN backlog,
# and no gate but this one sees it. (This exact bug shipped 2026-10-21 and
# left both gpu-* targets down beside a green deploy.)
mk_amd 0 42 1073741824 34359738368 "" 71000
port=$((19400 + RANDOM % 500))
python3 "$EXPORTER" --serve --amdgpu-sysfs "$SYS" --nvidia-cmd false \
--bind 127.0.0.1 --bind 127.0.0.2 --port "$port" \
> "$BATS_TEST_TMPDIR/serve.log" 2>&1 &
local pid=$!
trap 'kill "$pid" 2>/dev/null' EXIT
local ok=0
for _ in $(seq 1 20); do
curl -fsS -m2 "http://127.0.0.1:$port/metrics" >/dev/null 2>&1 && { ok=1; break; }
sleep 0.25
done
[ "$ok" -eq 1 ] || { kill "$pid" 2>/dev/null; return 1; }
# Both binds must ANSWER within 2 s each — the second one is the bug.
for b in 127.0.0.1 127.0.0.2; do
curl -fsS -m2 "http://$b:$port/metrics" | grep -q '^gpu_utilization_percent{' \
|| { echo "bind $b did not answer /metrics"; kill "$pid" 2>/dev/null; return 1; }
done
kill "$pid" 2>/dev/null
trap - EXIT
}