dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet

Browse Source 
Add a host-process GPU exporter, scrape it on both servers, and graph it
on the host-system dashboard. The fleet has four GPUs across three vendors
of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw)
and nothing exposed them until now.

scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs
(gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average)
plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus
families (utilization_percent, memory_used/total_bytes, temperature_celsius,
power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series,
never a fabricated 0: the R9700s expose no sysfs power, so they simply carry
no gpu_power_watts line. Both sources are injectable, so the seam runs on a
GPU-less dev box; live-verified on Dankserver's real sysfs.

It runs as a plain user unit (not a quadlet): it must read the host's own DRM
sysfs and shell to nvidia-smi, which a container cannot do without GPU
passthrough. Same posture as signal-relay. The base unit binds loopback; the
deploy drop-in appends the host LAN address, never a wildcard (matching
node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it
on both hosts and is wired into the planner (both targets), the test-changed
router, and two Prometheus scrape jobs using the node/cadvisor reach rules
(gateway on Dankclaw, LAN on Dankserver).
This commit is contained in:
Frank
2026-10-20 23:42:26 -07:00
parent b35269bfa5
commit b02647b992
9 changed files with 592 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Infrastructure/Observability.md
+22 -7
View File
@@ -2,7 +2,7 @@
type: reference
title: "Observability on Dankclaw"
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
timestamp: 2026-10-20
---
@@ -10,10 +10,10 @@ timestamp: 2026-10-20
## Summary
Seven containers and one host-side process: Prometheus (metrics), Grafana
Seven containers and two host-side processes: Prometheus (metrics), Grafana
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
carrying every firing group to Signal. All of it is Quadlet-managed,
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
boot-persistent and loopback-only, deployed by
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
end to end; the log half is configured, healthy, and carries no data.
@@ -35,6 +35,7 @@ that proves it today.
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
Every unit lives in `dankclaw/observability/quadlet/`, carries
@@ -45,10 +46,11 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
the same day, and `homepage-status.timer` — the root unit that probes all of
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
Dankserver runs two members of this family, node-exporter and cadvisor, as
its own units (`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`).
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
`gpu-exporter` (a user unit installed the same way as `signal-relay`).
## Exposure
@@ -117,6 +119,18 @@ verify it through the APIs.
`container` label. Network and filesystem are NOT exported for these containers
(no netns/classid attribution under the host-PID layout), so the dashboard
graphs only CPU, memory and OOM and does not fake the rest.
* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
`deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
`mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
°C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
`power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
a rootless Alloy cannot read the `0640` journal files, and the container-log
globs point at a root-storage layout that does not exist here. The fix is built
@@ -135,6 +149,7 @@ verify it through the APIs.
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)