feat(observability): per-container metrics via cadvisor + Containers dashboard
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a `containers` dashboard. cadvisor works on this rootless-Podman / cgroup-v2 host on one condition: --cgroupns=host. Podman starts every container in a private cgroup namespace, so without it cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the rootless app containers (it resolved 2 of 34), so naming comes from elsewhere: the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into a `container` label. No podman socket needed. cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 and the host LAN address — never a wildcard. Prometheus (a bridge container) reaches a host-published port at the LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports for these containers are graphed — CPU usage, memory working-set/limit, OOM — not network or filesystem, which are absent under the host-PID layout. Wired through the deployer (allowlist, deploy_observability case + /healthz gate, dispatcher), the planner (both targets), and observability_quadlet.bats.
This commit is contained in:
1 parent
961713e92d
commit
31908c623f
8 files changed
+398
-15
No files matched your search
@@ -0,0 +1,56 @@
|
||||
# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKCLAW.
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/cadvisor.container
|
||||
# Unit name: cadvisor.service (container name: cadvisor)
|
||||
#
|
||||
# WHY THIS EXISTS. The app stack is ~34 rootless Podman containers and until
|
||||
# now nothing reported per-container CPU/memory — Grafana could only see the
|
||||
# host (node_exporter) and the LLM engines. cadvisor reads the host cgroup
|
||||
# tree and exports container_cpu_* / container_memory_* per container.
|
||||
#
|
||||
# THE LOAD-BEARING FLAG IS --cgroupns=host. Podman starts every container in a
|
||||
# PRIVATE cgroup namespace, so an exporter container sees only its OWN cgroup
|
||||
# subtree: cadvisor exported a single root series until this flag was added,
|
||||
# then it walked the whole tree (996 per-container series measured 2026-10-20).
|
||||
# The quadlet unit name is carried in the cgroup path — app.slice/<name>.service/
|
||||
# libpod-payload-<hex> — so Prometheus relabels that path into a `container`
|
||||
# label (shared/prometheus/prometheus.yml, cadvisor-dankclaw job) and no
|
||||
# libpod/podman socket is needed to name containers.
|
||||
#
|
||||
# BINDING. cadvisor has no per-interface listen flag (unlike node_exporter's
|
||||
# --web.listen-address), so it binds 0.0.0.0 INSIDE its own network namespace.
|
||||
# The bridge PublishPort is what keeps the unauthenticated endpoint off the
|
||||
# host's other interfaces: it publishes exactly loopback (host-local probes and
|
||||
# the deployer's health gate) and this host's LAN address (Prometheus, a bridge
|
||||
# container, reaches a host-published port at its LAN address — the same netavark
|
||||
# rule node-exporter relies on, only here the port really is published).
|
||||
#
|
||||
# It is rootless on purpose: dank owns the app containers' cgroups, so a
|
||||
# rootless cadvisor reads every one of them; root would buy nothing here.
|
||||
|
||||
[Unit]
|
||||
Description=cAdvisor container metrics (Dankclaw)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=cadvisor
|
||||
Image=gcr.io/cadvisor/cadvisor:v0.49.1
|
||||
PodmanArgs=--pid=host --cgroupns=host
|
||||
PublishPort=127.0.0.1:8080:8080
|
||||
PublishPort=192.168.0.143:8080:8080
|
||||
Volume=/sys:/sys:ro
|
||||
Volume=/:/rootfs:ro
|
||||
Exec=--housekeeping_interval=30s
|
||||
HealthCmd=wget --spider -q http://localhost:8080/healthz
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=40s
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
Reference in new issue
Block a user