feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet
Add a host-process GPU exporter, scrape it on both servers, and graph it on the host-system dashboard. The fleet has four GPUs across three vendors of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw) and nothing exposed them until now. scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs (gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average) plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus families (utilization_percent, memory_used/total_bytes, temperature_celsius, power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series, never a fabricated 0: the R9700s expose no sysfs power, so they simply carry no gpu_power_watts line. Both sources are injectable, so the seam runs on a GPU-less dev box; live-verified on Dankserver's real sysfs. It runs as a plain user unit (not a quadlet): it must read the host's own DRM sysfs and shell to nvidia-smi, which a container cannot do without GPU passthrough. Same posture as signal-relay. The base unit binds loopback; the deploy drop-in appends the host LAN address, never a wildcard (matching node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it on both hosts and is wired into the planner (both targets), the test-changed router, and two Prometheus scrape jobs using the node/cadvisor reach rules (gateway on Dankclaw, LAN on Dankserver).
This commit is contained in:
1 parent
b35269bfa5
commit
b02647b992
9 files changed
+592
-9
No files matched your search
@@ -191,3 +191,25 @@ scrape_configs:
|
||||
action: keep
|
||||
- regex: 'id|name|image|container_label.*|machine_id|boot_id'
|
||||
action: labeldrop
|
||||
|
||||
# Per-GPU metrics (gpu_utilization_percent / gpu_memory_used_bytes /
|
||||
# gpu_memory_total_bytes / gpu_temperature_celsius / gpu_power_watts) from the
|
||||
# host-process gpu-exporter (scripts/gpu-exporter.py). One job per host, same
|
||||
# reach rules as node/cadvisor: Dankclaw is scraped at the podman gateway
|
||||
# because the netavark hairpin quirk makes Prometheus fail to dial its own
|
||||
# host LAN IP; Dankserver is plain LAN. The gpu label (PCI BDF on AMD, index on
|
||||
# NVIDIA) + vendor/model come from the exporter, so the GPU row on the
|
||||
# host-system dashboard filters by host. Power is emitted only where readable
|
||||
# (NVIDIA + the Dankclaw iGPU); the R9700s have no sysfs power, so they simply
|
||||
# have no gpu_power_watts series — no fabricated zero.
|
||||
- job_name: "gpu-dankclaw"
|
||||
static_configs:
|
||||
- targets: ["host.containers.internal:9401"]
|
||||
labels:
|
||||
host: dankclaw
|
||||
|
||||
- job_name: "gpu-dankserver"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.235:9401"]
|
||||
labels:
|
||||
host: dankserver
|
||||
Reference in new issue
Block a user