feat(observability): per-container metrics via cadvisor + Containers dashboard
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a `containers` dashboard. cadvisor works on this rootless-Podman / cgroup-v2 host on one condition: --cgroupns=host. Podman starts every container in a private cgroup namespace, so without it cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the rootless app containers (it resolved 2 of 34), so naming comes from elsewhere: the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into a `container` label. No podman socket needed. cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 and the host LAN address — never a wildcard. Prometheus (a bridge container) reaches a host-published port at the LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports for these containers are graphed — CPU usage, memory working-set/limit, OOM — not network or filesystem, which are absent under the host-PID layout. Wired through the deployer (allowlist, deploy_observability case + /healthz gate, dispatcher), the planner (both targets), and observability_quadlet.bats.
This commit is contained in:
@@ -39,6 +39,7 @@ SERVICE_ORDER = (
|
||||
"loki",
|
||||
"alloy",
|
||||
"node-exporter",
|
||||
"cadvisor",
|
||||
"alertmanager",
|
||||
"authentik-postgres",
|
||||
"authentik-server",
|
||||
@@ -92,6 +93,7 @@ DANKCLAW_SERVICES = {
|
||||
"loki",
|
||||
"alloy",
|
||||
"node-exporter",
|
||||
"cadvisor",
|
||||
"alertmanager",
|
||||
"authentik-postgres",
|
||||
"authentik-server",
|
||||
@@ -124,6 +126,7 @@ DANKSERVER_SERVICES = {
|
||||
"readarr",
|
||||
"frank-archon",
|
||||
"node-exporter",
|
||||
"cadvisor",
|
||||
}
|
||||
|
||||
# FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
|
||||
@@ -374,7 +377,7 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# consumer (prometheus before grafana, loki before alloy).
|
||||
if len(parts) >= 4 and parts[2] == "quadlet":
|
||||
name = parts[3].removesuffix(".container")
|
||||
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
|
||||
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager"}:
|
||||
_add(services, name)
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "gitea":
|
||||
@@ -417,8 +420,8 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# node-dankserver job had nothing to scrape.
|
||||
if len(parts) >= 4 and parts[2] == "quadlet":
|
||||
name = parts[3].removesuffix(".container")
|
||||
if name == "node-exporter":
|
||||
_add(services, "node-exporter")
|
||||
if name in {"node-exporter", "cadvisor"}:
|
||||
_add(services, name)
|
||||
return services
|
||||
# frank-archon's quadlet-era directory left the tree on 2026-09-26
|
||||
# (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
|
||||
|
||||
Reference in New Issue
Block a user