Frank
31908c623f
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a `containers` dashboard. cadvisor works on this rootless-Podman / cgroup-v2 host on one condition: --cgroupns=host. Podman starts every container in a private cgroup namespace, so without it cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the rootless app containers (it resolved 2 of 34), so naming comes from elsewhere: the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into a `container` label. No podman socket needed. cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 and the host LAN address — never a wildcard. Prometheus (a bridge container) reaches a host-published port at the LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports for these containers are graphed — CPU usage, memory working-set/limit, OOM — not network or filesystem, which are absent under the host-PID layout. Wired through the deployer (allowlist, deploy_observability case + /healthz gate, dispatcher), the planner (both targets), and observability_quadlet.bats.
57 lines
2.3 KiB
INI
57 lines
2.3 KiB
INI
# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKCLAW.
|
|
#
|
|
# Install path on dankclaw: ~/.config/containers/systemd/cadvisor.container
|
|
# Unit name: cadvisor.service (container name: cadvisor)
|
|
#
|
|
# WHY THIS EXISTS. The app stack is ~34 rootless Podman containers and until
|
|
# now nothing reported per-container CPU/memory — Grafana could only see the
|
|
# host (node_exporter) and the LLM engines. cadvisor reads the host cgroup
|
|
# tree and exports container_cpu_* / container_memory_* per container.
|
|
#
|
|
# THE LOAD-BEARING FLAG IS --cgroupns=host. Podman starts every container in a
|
|
# PRIVATE cgroup namespace, so an exporter container sees only its OWN cgroup
|
|
# subtree: cadvisor exported a single root series until this flag was added,
|
|
# then it walked the whole tree (996 per-container series measured 2026-10-20).
|
|
# The quadlet unit name is carried in the cgroup path — app.slice/<name>.service/
|
|
# libpod-payload-<hex> — so Prometheus relabels that path into a `container`
|
|
# label (shared/prometheus/prometheus.yml, cadvisor-dankclaw job) and no
|
|
# libpod/podman socket is needed to name containers.
|
|
#
|
|
# BINDING. cadvisor has no per-interface listen flag (unlike node_exporter's
|
|
# --web.listen-address), so it binds 0.0.0.0 INSIDE its own network namespace.
|
|
# The bridge PublishPort is what keeps the unauthenticated endpoint off the
|
|
# host's other interfaces: it publishes exactly loopback (host-local probes and
|
|
# the deployer's health gate) and this host's LAN address (Prometheus, a bridge
|
|
# container, reaches a host-published port at its LAN address — the same netavark
|
|
# rule node-exporter relies on, only here the port really is published).
|
|
#
|
|
# It is rootless on purpose: dank owns the app containers' cgroups, so a
|
|
# rootless cadvisor reads every one of them; root would buy nothing here.
|
|
|
|
[Unit]
|
|
Description=cAdvisor container metrics (Dankclaw)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=cadvisor
|
|
Image=gcr.io/cadvisor/cadvisor:v0.49.1
|
|
PodmanArgs=--pid=host --cgroupns=host
|
|
PublishPort=127.0.0.1:8080:8080
|
|
PublishPort=192.168.0.143:8080:8080
|
|
Volume=/sys:/sys:ro
|
|
Volume=/:/rootfs:ro
|
|
Exec=--housekeeping_interval=30s
|
|
HealthCmd=wget --spider -q http://localhost:8080/healthz
|
|
HealthInterval=30s
|
|
HealthTimeout=10s
|
|
HealthRetries=3
|
|
HealthStartPeriod=40s
|
|
|
|
[Service]
|
|
Restart=always
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|