feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet
Add a host-process GPU exporter, scrape it on both servers, and graph it on the host-system dashboard. The fleet has four GPUs across three vendors of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw) and nothing exposed them until now. scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs (gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average) plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus families (utilization_percent, memory_used/total_bytes, temperature_celsius, power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series, never a fabricated 0: the R9700s expose no sysfs power, so they simply carry no gpu_power_watts line. Both sources are injectable, so the seam runs on a GPU-less dev box; live-verified on Dankserver's real sysfs. It runs as a plain user unit (not a quadlet): it must read the host's own DRM sysfs and shell to nvidia-smi, which a container cannot do without GPU passthrough. Same posture as signal-relay. The base unit binds loopback; the deploy drop-in appends the host LAN address, never a wildcard (matching node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it on both hosts and is wired into the planner (both targets), the test-changed router, and two Prometheus scrape jobs using the node/cadvisor reach rules (gateway on Dankclaw, LAN on Dankserver).
This commit is contained in:
1 parent
b35269bfa5
commit
b02647b992
9 files changed
+592
-9
No files matched your search
@@ -40,6 +40,7 @@ SERVICE_ORDER = (
|
||||
"alloy",
|
||||
"node-exporter",
|
||||
"cadvisor",
|
||||
"gpu-exporter",
|
||||
"alertmanager",
|
||||
"authentik-postgres",
|
||||
"authentik-server",
|
||||
@@ -94,6 +95,7 @@ DANKCLAW_SERVICES = {
|
||||
"alloy",
|
||||
"node-exporter",
|
||||
"cadvisor",
|
||||
"gpu-exporter",
|
||||
"alertmanager",
|
||||
"authentik-postgres",
|
||||
"authentik-server",
|
||||
@@ -127,6 +129,7 @@ DANKSERVER_SERVICES = {
|
||||
"frank-archon",
|
||||
"node-exporter",
|
||||
"cadvisor",
|
||||
"gpu-exporter",
|
||||
}
|
||||
|
||||
# FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
|
||||
@@ -452,6 +455,12 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# change to it redeploying alertmanager is the only path that
|
||||
# reaches it on the host.
|
||||
_add(services, "alertmanager")
|
||||
if name in {"gpu-exporter.py", "gpu-exporter.service"}:
|
||||
# The host-process GPU exporter is installed directly by the
|
||||
# gpu-exporter deploy on BOTH hosts, so its script and unit select
|
||||
# that service (target_services then keeps it only on the host being
|
||||
# planned). Editing the collector restarts the running exporter.
|
||||
_add(services, "gpu-exporter")
|
||||
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
|
||||
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-zealot only. The ik_llama.cpp
|
||||
|
||||
Reference in new issue
Block a user