feat(observability): per-container metrics via cadvisor + Containers dashboard

Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.

cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.

cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.

Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
This commit is contained in:
Frank
2026-10-20 22:51:42 -07:00
parent 961713e92d
commit 31908c623f
8 changed files with 398 additions and 15 deletions
+6 -3
View File
@@ -39,6 +39,7 @@ SERVICE_ORDER = (
"loki",
"alloy",
"node-exporter",
"cadvisor",
"alertmanager",
"authentik-postgres",
"authentik-server",
@@ -92,6 +93,7 @@ DANKCLAW_SERVICES = {
"loki",
"alloy",
"node-exporter",
"cadvisor",
"alertmanager",
"authentik-postgres",
"authentik-server",
@@ -124,6 +126,7 @@ DANKSERVER_SERVICES = {
"readarr",
"frank-archon",
"node-exporter",
"cadvisor",
}
# FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
@@ -374,7 +377,7 @@ def services_for_path(path: str, target: str) -> set[str]:
# consumer (prometheus before grafana, loki before alloy).
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3].removesuffix(".container")
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager"}:
_add(services, name)
return services
if len(parts) >= 2 and parts[1] == "gitea":
@@ -417,8 +420,8 @@ def services_for_path(path: str, target: str) -> set[str]:
# node-dankserver job had nothing to scrape.
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3].removesuffix(".container")
if name == "node-exporter":
_add(services, "node-exporter")
if name in {"node-exporter", "cadvisor"}:
_add(services, name)
return services
# frank-archon's quadlet-era directory left the tree on 2026-09-26
# (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —