fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and cadvisor-dankclaw DOWN beside a green deploy: - gpu-exporter.py built one ThreadingHTTPServer per bind but ran serve_forever() on servers[0] only: the LAN-class socket sat listen()ing but never accept()ed, loopback deploy gate passed, every scrape that reached the other bind timed out on the SYN backlog. Now one thread per extra bind. Regression test spins --serve on two loopback binds and requires BOTH to answer /metrics. - cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the Prometheus container: a bridge container cannot dial its own host's LAN address at all (hairpin refuses published ports too); the podman gateway answers. Target moved to host.containers.internal:8080; the seam that pinned the LAN address (asserting the wrong doctrine) is re-pinned to the gateway plus a negative pin against the own-host LAN dial. Observability page records the measured reachability law replacing the inferred one.
This commit is contained in:
1 parent
b62a05837f
commit
650b6e0281
5 files changed
+68
-15
No files matched your search
@@ -140,12 +140,15 @@ scrape_configs:
|
||||
# system.slice daemons and machine.slice scopes carry no such name and are
|
||||
# dropped, as is the per-CPU-core split (only cpu="total" survives), so the
|
||||
# scrape stays small. cadvisor cannot bind per-interface, so it is reached at
|
||||
# the host's PUBLISHED LAN port: unlike node-exporter's host-net bind (which
|
||||
# is why node-dankclaw scrapes via host.containers.internal), a published port
|
||||
# IS reachable from this bridge container at the host LAN address.
|
||||
# the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
|
||||
# DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
|
||||
# connection whether the port is a host-net bind or a published port (the
|
||||
# original note here claimed published ports were exempt; measured 2026-10-21
|
||||
# from inside this container: LAN refused, gateway answers). Cross-host
|
||||
# targets (dankserver) have no hairpin problem and stay plain LAN.
|
||||
- job_name: "cadvisor-dankclaw"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.143:8080"]
|
||||
- targets: ["host.containers.internal:8080"]
|
||||
labels:
|
||||
host: dankclaw
|
||||
metric_relabel_configs:
|
||||
|
||||
Reference in new issue
Block a user