Both GPU targets and cadvisor-dankclaw sit DOWN in Prometheus beside a green deploy (found while answering "I can't see GPU stats"). Two independent bugs, both shipped 2026-10-20.
1. gpu-exporter served only its first bind
main() built one ThreadingHTTPServer per declared bind but called serve_forever() on servers[0] only. The LAN-class socket sat listen()ing — visible in ss, backlog filling — and never accepted: the loopback deploy gate passed in 0.59 s while every scrape reaching the other bind died on the SYN backlog. Hand-proved red before the fix: bind1 200, bind2 timeout.
Fix: one daemon thread per extra bind. New seam test spins --serve on two loopback binds and requires both to answer /metrics — the contract the old suite never pinned (it smoke-tested a single bind).
2. cadvisor-dankclaw dialed its own host's LAN IP
Measured from inside the Prometheus container:
169.254.1.2:9100 -> 200 (node, via gateway)
169.254.1.2:8080 -> ok (cadvisor, via gateway)
192.168.0.143:8080 -> refused (own-host LAN — hairpin, published port or not)
A bridge container cannot dial its own host's LAN address at all; the netavark hairpin refuses published ports exactly like host-net binds — the inference the cadvisor PR carried ("published port IS reachable at the LAN address") was wrong, and the seam pinned the broken address as doctrine. Target now host.containers.internal:8080; the seam re-pinned to the gateway with a negative pin against the own-host LAN dial. Cross-host (dankserver) stays plain LAN — no hairpin there.
Observability.md now carries the measured same-host reachability law in place of the inferred one.
Verification
Red proof by hand (bind timeout) before the fix; new test green after.
## What
Both GPU targets and cadvisor-dankclaw sit **DOWN** in Prometheus beside a green deploy (found while answering "I can't see GPU stats"). Two independent bugs, both shipped 2026-10-20.
### 1. gpu-exporter served only its first bind
`main()` built one `ThreadingHTTPServer` per declared bind but called `serve_forever()` on `servers[0]` only. The LAN-class socket sat `listen()`ing — visible in `ss`, backlog filling — and **never accepted**: the loopback deploy gate passed in 0.59 s while every scrape reaching the other bind died on the SYN backlog. Hand-proved red before the fix: bind1 `200`, bind2 timeout.
Fix: one daemon thread per extra bind. New seam test spins `--serve` on **two** loopback binds and requires **both** to answer `/metrics` — the contract the old suite never pinned (it smoke-tested a single bind).
### 2. cadvisor-dankclaw dialed its own host's LAN IP
Measured from inside the Prometheus container:
```
169.254.1.2:9100 -> 200 (node, via gateway)
169.254.1.2:8080 -> ok (cadvisor, via gateway)
192.168.0.143:8080 -> refused (own-host LAN — hairpin, published port or not)
```
A bridge container cannot dial its own host's LAN address at all; the netavark hairpin refuses **published ports exactly like host-net binds** — the inference the cadvisor PR carried ("published port IS reachable at the LAN address") was wrong, and the seam **pinned the broken address as doctrine**. Target now `host.containers.internal:8080`; the seam re-pinned to the gateway with a **negative pin** against the own-host LAN dial. Cross-host (dankserver) stays plain LAN — no hairpin there.
`Observability.md` now carries the measured same-host reachability law in place of the inferred one.
## Verification
- Red proof by hand (bind timeout) before the fix; new test green after.
- `./tests/run.sh` — 490 passed.
---
cwd: /tmp/wt-gpufix · session: 01a15b5c-ede9-7063-b05b-1ef94188b7c3
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:
- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
but never accept()ed, loopback deploy gate passed, every scrape that
reached the other bind timed out on the SYN backlog. Now one thread per
extra bind. Regression test spins --serve on two loopback binds and
requires BOTH to answer /metrics.
- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
Prometheus container: a bridge container cannot dial its own host's LAN
address at all (hairpin refuses published ports too); the podman gateway
answers. Target moved to host.containers.internal:8080; the seam that
pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
gateway plus a negative pin against the own-host LAN dial. Observability
page records the measured reachability law replacing the inferred one.
dank
merged commit fbeb9701e4 into main2026-10-21 00:39:54 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
Both GPU targets and cadvisor-dankclaw sit DOWN in Prometheus beside a green deploy (found while answering "I can't see GPU stats"). Two independent bugs, both shipped 2026-10-20.
1. gpu-exporter served only its first bind
main()built oneThreadingHTTPServerper declared bind but calledserve_forever()onservers[0]only. The LAN-class socket satlisten()ing — visible inss, backlog filling — and never accepted: the loopback deploy gate passed in 0.59 s while every scrape reaching the other bind died on the SYN backlog. Hand-proved red before the fix: bind1200, bind2 timeout.Fix: one daemon thread per extra bind. New seam test spins
--serveon two loopback binds and requires both to answer/metrics— the contract the old suite never pinned (it smoke-tested a single bind).2. cadvisor-dankclaw dialed its own host's LAN IP
Measured from inside the Prometheus container:
A bridge container cannot dial its own host's LAN address at all; the netavark hairpin refuses published ports exactly like host-net binds — the inference the cadvisor PR carried ("published port IS reachable at the LAN address") was wrong, and the seam pinned the broken address as doctrine. Target now
host.containers.internal:8080; the seam re-pinned to the gateway with a negative pin against the own-host LAN dial. Cross-host (dankserver) stays plain LAN — no hairpin there.Observability.mdnow carries the measured same-host reachability law in place of the inferred one.Verification
./tests/run.sh— 490 passed.cwd: /tmp/wt-gpufix · session: 01a15b5c-ede9-7063-b05b-1ef94188b7c3