dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(monitoring): scrape node-dankclaw via host.containers.internal

Browse Source 
node-exporter is net=host, so its 192.168.0.143:9100 bind is a host bind,
not a published port — the bridge-networked Prometheus container cannot
reach its own host's LAN IP there (netavark DNATs published ports only).
The node-dankclaw target has been red by construction since the
observability cutover while the container reported healthy. Point it at
host.containers.internal:9100 (netavark delivers that to the host LAN IP,
where node-exporter listens) — the same rule frank-large and frank-small
already use. node-dankserver is a different machine and keeps its LAN
address. Update the cross-file contract test and the Targets table.
This commit is contained in:
Frank
2026-09-21 11:09:03 -07:00
parent f998a2d9f2
commit 5c1006dda6
3 changed files with 31 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Infrastructure/Prometheus.md
+4 -2
View File
@@ -25,9 +25,11 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| Job | Target | Why |
|-----|--------|-----|
| `prometheus` | `localhost:9090` | itself |
| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | host metrics from each machine's [[Node-Exporter]] |
| `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
| `frank-medium`, `frank-large` | `192.168.0.235:7770`, `192.168.0.143:7770` | the LLM fleet — `vllm:` and `llamacpp:` series (LAN addresses, not the guarded vhosts) |
| `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
| `frank-large`, `frank-small` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
**Three jobs read down, and they are not outages** (audited 2026-09-16):