Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:
- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
but never accept()ed, loopback deploy gate passed, every scrape that
reached the other bind timed out on the SYN backlog. Now one thread per
extra bind. Regression test spins --serve on two loopback binds and
requires BOTH to answer /metrics.
- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
Prometheus container: a bridge container cannot dial its own host's LAN
address at all (hairpin refuses published ports too); the podman gateway
answers. Target moved to host.containers.internal:8080; the seam that
pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
gateway plus a negative pin against the own-host LAN dial. Observability
page records the measured reachability law replacing the inferred one.
Add a host-process GPU exporter, scrape it on both servers, and graph it
on the host-system dashboard. The fleet has four GPUs across three vendors
of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw)
and nothing exposed them until now.
scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs
(gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average)
plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus
families (utilization_percent, memory_used/total_bytes, temperature_celsius,
power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series,
never a fabricated 0: the R9700s expose no sysfs power, so they simply carry
no gpu_power_watts line. Both sources are injectable, so the seam runs on a
GPU-less dev box; live-verified on Dankserver's real sysfs.
It runs as a plain user unit (not a quadlet): it must read the host's own DRM
sysfs and shell to nvidia-smi, which a container cannot do without GPU
passthrough. Same posture as signal-relay. The base unit binds loopback; the
deploy drop-in appends the host LAN address, never a wildcard (matching
node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it
on both hosts and is wired into the planner (both targets), the test-changed
router, and two Prometheus scrape jobs using the node/cadvisor reach rules
(gateway on Dankclaw, LAN on Dankserver).
Frank