Per-GPU metrics for the mixed AMD/NVIDIA fleet — a host-process exporter, Prometheus scrape jobs on both servers, and a GPU block on the host-system dashboard. The four GPUs (2x R9700 @ Dankserver, 8060S iGPU + RTX 3090 @ Dankclaw) were invisible to the stack until now.
How
scripts/gpu-exporter.py — reads each AMD card's amdgpu sysfs (gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average) plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* families (util %, VRAM used/total bytes, temp °C, power W) labelled gpu/vendor/model, with host from the scrape job. Absent hardware yields absent series, never a fabricated 0 — the R9700s expose no sysfs power, so they carry no gpu_power_watts line.
Runs as a user unit, not a quadlet: it must read the host's own DRM sysfs and shell to nvidia-smi, which a container cannot do without GPU passthrough. Same posture as signal-relay. Base unit binds loopback; the deploy drop-in appends the host LAN address — never a wildcard (matching node-exporter/cadvisor).
deploy_gpu_exporter installs/enables/health-gates it on both hosts, and is wired into the planner (both targets), the test-changed router, and two Prometheus scrape jobs using the node/cadvisor reach rules (podman gateway on Dankclaw, LAN on Dankserver).
Evidence
tests/gpu_exporter.bats — 12 tests: the collector's black-box contract (fixture sysfs + injected nvidia-smi stub, incl. absent-power and failing-nvidia arms), plus the unit / deployer / planner / Prometheus / dashboard cross-file contracts.
Live on Dankserver: gpu-exporter.py --serve answered /metrics with all three cards, 404 on unknown paths, freed the port on kill.
./tests/run.sh — all 488 tests pass.
Deploy
Live after deploy.sh <host> --service gpu-exporter on each host, plus a prometheus + grafana deploy to pick up the scrape jobs and the dashboard. Not merged/deployed yet — awaiting your OK.
## What
Per-GPU metrics for the mixed AMD/NVIDIA fleet — a host-process exporter, Prometheus scrape jobs on both servers, and a GPU block on the `host-system` dashboard. The four GPUs (2x R9700 @ Dankserver, 8060S iGPU + RTX 3090 @ Dankclaw) were invisible to the stack until now.
## How
- `scripts/gpu-exporter.py` — reads each AMD card's amdgpu sysfs (`gpu_busy_percent` / `mem_info_vram_*` / `hwmon temp1_input` / `power1_average`) plus one `nvidia-smi --query-gpu` CSV, and emits unified `gpu_*` families (util %, VRAM used/total bytes, temp °C, power W) labelled `gpu`/`vendor`/`model`, with `host` from the scrape job. **Absent hardware yields absent series, never a fabricated 0** — the R9700s expose no sysfs power, so they carry no `gpu_power_watts` line.
- Runs as a **user unit, not a quadlet**: it must read the host's own DRM sysfs and shell to `nvidia-smi`, which a container cannot do without GPU passthrough. Same posture as `signal-relay`. Base unit binds loopback; the deploy drop-in appends the host LAN address — never a wildcard (matching node-exporter/cadvisor).
- `deploy_gpu_exporter` installs/enables/health-gates it on both hosts, and is wired into the planner (both targets), the `test-changed` router, and two Prometheus scrape jobs using the node/cadvisor reach rules (podman gateway on Dankclaw, LAN on Dankserver).
## Evidence
- `tests/gpu_exporter.bats` — 12 tests: the collector's black-box contract (fixture sysfs + injected `nvidia-smi` stub, incl. absent-power and failing-nvidia arms), plus the unit / deployer / planner / Prometheus / dashboard cross-file contracts.
- Live on Dankserver: `gpu-exporter.py --serve` answered `/metrics` with all three cards, 404 on unknown paths, freed the port on kill.
- `./tests/run.sh` — all 488 tests pass.
## Deploy
Live after `deploy.sh <host> --service gpu-exporter` on each host, plus a `prometheus` + `grafana` deploy to pick up the scrape jobs and the dashboard. Not merged/deployed yet — awaiting your OK.
---
cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 01a15b5c-ede9-7063-b05b-1ef94188b7c3
(build worktree: /tmp/wt-gpu)
Add a host-process GPU exporter, scrape it on both servers, and graph it
on the host-system dashboard. The fleet has four GPUs across three vendors
of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw)
and nothing exposed them until now.
scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs
(gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average)
plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus
families (utilization_percent, memory_used/total_bytes, temperature_celsius,
power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series,
never a fabricated 0: the R9700s expose no sysfs power, so they simply carry
no gpu_power_watts line. Both sources are injectable, so the seam runs on a
GPU-less dev box; live-verified on Dankserver's real sysfs.
It runs as a plain user unit (not a quadlet): it must read the host's own DRM
sysfs and shell to nvidia-smi, which a container cannot do without GPU
passthrough. Same posture as signal-relay. The base unit binds loopback; the
deploy drop-in appends the host LAN address, never a wildcard (matching
node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it
on both hosts and is wired into the planner (both targets), the test-changed
router, and two Prometheus scrape jobs using the node/cadvisor reach rules
(gateway on Dankclaw, LAN on Dankserver).
dank
merged commit a2011f4eb6 into main2026-10-20 23:50:11 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
Per-GPU metrics for the mixed AMD/NVIDIA fleet — a host-process exporter, Prometheus scrape jobs on both servers, and a GPU block on the
host-systemdashboard. The four GPUs (2x R9700 @ Dankserver, 8060S iGPU + RTX 3090 @ Dankclaw) were invisible to the stack until now.How
scripts/gpu-exporter.py— reads each AMD card's amdgpu sysfs (gpu_busy_percent/mem_info_vram_*/hwmon temp1_input/power1_average) plus onenvidia-smi --query-gpuCSV, and emits unifiedgpu_*families (util %, VRAM used/total bytes, temp °C, power W) labelledgpu/vendor/model, withhostfrom the scrape job. Absent hardware yields absent series, never a fabricated 0 — the R9700s expose no sysfs power, so they carry nogpu_power_wattsline.nvidia-smi, which a container cannot do without GPU passthrough. Same posture assignal-relay. Base unit binds loopback; the deploy drop-in appends the host LAN address — never a wildcard (matching node-exporter/cadvisor).deploy_gpu_exporterinstalls/enables/health-gates it on both hosts, and is wired into the planner (both targets), thetest-changedrouter, and two Prometheus scrape jobs using the node/cadvisor reach rules (podman gateway on Dankclaw, LAN on Dankserver).Evidence
tests/gpu_exporter.bats— 12 tests: the collector's black-box contract (fixture sysfs + injectednvidia-smistub, incl. absent-power and failing-nvidia arms), plus the unit / deployer / planner / Prometheus / dashboard cross-file contracts.gpu-exporter.py --serveanswered/metricswith all three cards, 404 on unknown paths, freed the port on kill../tests/run.sh— all 488 tests pass.Deploy
Live after
deploy.sh <host> --service gpu-exporteron each host, plus aprometheus+grafanadeploy to pick up the scrape jobs and the dashboard. Not merged/deployed yet — awaiting your OK.cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 01a15b5c-ede9-7063-b05b-1ef94188b7c3
(build worktree: /tmp/wt-gpu)