Frank
b02647b992
Add a host-process GPU exporter, scrape it on both servers, and graph it on the host-system dashboard. The fleet has four GPUs across three vendors of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw) and nothing exposed them until now. scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs (gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average) plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus families (utilization_percent, memory_used/total_bytes, temperature_celsius, power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series, never a fabricated 0: the R9700s expose no sysfs power, so they simply carry no gpu_power_watts line. Both sources are injectable, so the seam runs on a GPU-less dev box; live-verified on Dankserver's real sysfs. It runs as a plain user unit (not a quadlet): it must read the host's own DRM sysfs and shell to nvidia-smi, which a container cannot do without GPU passthrough. Same posture as signal-relay. The base unit binds loopback; the deploy drop-in appends the host LAN address, never a wildcard (matching node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it on both hosts and is wired into the planner (both targets), the test-changed router, and two Prometheus scrape jobs using the node/cadvisor reach rules (gateway on Dankclaw, LAN on Dankserver).
36 lines
1.6 KiB
Desktop File
36 lines
1.6 KiB
Desktop File
# Systemd USER unit: gpu-exporter (both Dankclaw and Dankserver).
|
|
#
|
|
# Install path: ~/.config/systemd/user/gpu-exporter.service
|
|
# Installed and enabled by deploy_gpu_exporter in scripts/deploy-remote.sh.
|
|
#
|
|
# Per-GPU metrics (utilization / VRAM / temperature / power) for the mixed
|
|
# AMD + NVIDIA fleet, scraped by Prometheus. Like signal-relay it is a plain
|
|
# user unit, not a quadlet: it runs python3 straight from the checkout (a pull
|
|
# updates it with no copy step), a hand-written user unit can be enabled where
|
|
# the podman v1 generator's units cannot, and with Linger=yes for dank it starts
|
|
# at boot. It reads the host's own cgroup/drm sysfs and shells to nvidia-smi, so
|
|
# it must run on the host — a container would need GPU/DRM passthrough for
|
|
# nothing gained.
|
|
#
|
|
# BINDING. The base unit serves loopback only. deploy_gpu_exporter writes a
|
|
# drop-in adding THIS host's LAN address (the same two-address posture
|
|
# node-exporter and cadvisor use), so the endpoint is never a wildcard. It is
|
|
# unauthenticated metrics on the LAN, identical to node-exporter :9100 and
|
|
# cadvisor :8080 — Prometheus reaches same-host via host.containers.internal and
|
|
# the Dankserver instance over the LAN from Dankclaw.
|
|
|
|
[Unit]
|
|
Description=GPU metrics exporter (amdgpu sysfs + nvidia-smi)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
# Loopback by default; the deploy drop-in appends the host LAN address.
|
|
Environment=GPU_EXPORTER_BINDS=127.0.0.1
|
|
ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve
|
|
Restart=always
|
|
RestartSec=5
|
|
|
|
[Install]
|
|
WantedBy=default.target
|