feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet

Add a host-process GPU exporter, scrape it on both servers, and graph it
on the host-system dashboard. The fleet has four GPUs across three vendors
of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw)
and nothing exposed them until now.

scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs
(gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average)
plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus
families (utilization_percent, memory_used/total_bytes, temperature_celsius,
power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series,
never a fabricated 0: the R9700s expose no sysfs power, so they simply carry
no gpu_power_watts line. Both sources are injectable, so the seam runs on a
GPU-less dev box; live-verified on Dankserver's real sysfs.

It runs as a plain user unit (not a quadlet): it must read the host's own DRM
sysfs and shell to nvidia-smi, which a container cannot do without GPU
passthrough. Same posture as signal-relay. The base unit binds loopback; the
deploy drop-in appends the host LAN address, never a wildcard (matching
node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it
on both hosts and is wired into the planner (both targets), the test-changed
router, and two Prometheus scrape jobs using the node/cadvisor reach rules
(gateway on Dankclaw, LAN on Dankserver).
This commit is contained in:
Frank committed 2026-10-20 23:42:26 -07:00
1 parent b35269bfa5
commit b02647b992
9 files changed
+592 -9

No files matched your search

+8
View File
@@ -293,6 +293,14 @@ for path in "${CHANGED[@]}"; do
add_test tests/caddyfile.bats
add_test tests/deployment_scoping.bats
;;
# The GPU exporter is a host process; its seam pins the collector, the
# user unit, the deployer function, the planner route, the Prometheus
# scrape jobs and the host-system GPU panels — a cross-file contract, so
# the whole seam rides on any of its own files.
scripts/gpu-exporter.py|scripts/gpu-exporter.service)
add_test tests/gpu_exporter.bats
add_test tests/deployment_scoping.bats
;;
dankclaw/udev/*)
# Host-level udev rules are applied by manual sudo install + udevadm
# trigger, not dispatched as a service; nothing here is bats-testable.