feat(observability): per-container metrics via cadvisor + Containers dashboard

Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.

cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.

cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.

Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
This commit is contained in:
Frank committed 2026-10-20 22:51:42 -07:00
1 parent 961713e92d
commit 31908c623f
8 files changed
+398 -15

No files matched your search

+60
View File
@@ -131,3 +131,63 @@ scrape_configs:
- targets: ["192.168.0.235:7770"]
labels:
service: frank-archon
# Per-container metrics from cadvisor (a rootless quadlet on each host).
# cadvisor keys every series by the cgroup path; for a quadlet container that
# path is …/app.slice/<name>.service[/libpod-payload-<hex>], so the unit NAME
# lives in it. The relabel below pulls that name out into a `container` label
# and keeps only the per-unit aggregate — cadvisor's raw host root,
# system.slice daemons and machine.slice scopes carry no such name and are
# dropped, as is the per-CPU-core split (only cpu="total" survives), so the
# scrape stays small. cadvisor cannot bind per-interface, so it is reached at
# the host's PUBLISHED LAN port: unlike node-exporter's host-net bind (which
# is why node-dankclaw scrapes via host.containers.internal), a published port
# IS reachable from this bridge container at the host LAN address.
- job_name: "cadvisor-dankclaw"
static_configs:
- targets: ["192.168.0.143:8080"]
labels:
host: dankclaw
metric_relabel_configs:
- source_labels: [id]
regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
target_label: container
replacement: '$1'
# Keep only the unit-level cgroup; drop its payload/control children so a
# container is not counted twice.
- source_labels: [id]
regex: '.*/\.service/.*'
action: drop
# Drop everything that is not a named unit (no `container` was extracted).
- source_labels: [container]
regex: '.+'
action: keep
# cpu="total" for cpu series; metrics with no cpu label (memory, network,
# filesystem) are kept.
- source_labels: [cpu]
regex: '^$|total'
action: keep
- regex: 'id|name|image|container_label.*|machine_id|boot_id'
action: labeldrop
- job_name: "cadvisor-dankserver"
static_configs:
- targets: ["192.168.0.235:8080"]
labels:
host: dankserver
metric_relabel_configs:
- source_labels: [id]
regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
target_label: container
replacement: '$1'
- source_labels: [id]
regex: '.*/\.service/.*'
action: drop
- source_labels: [container]
regex: '.+'
action: keep
- source_labels: [cpu]
regex: '^$|total'
action: keep
- regex: 'id|name|image|container_label.*|machine_id|boot_id'
action: labeldrop