feat(observability): per-container metrics via cadvisor + Containers dashboard
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a `containers` dashboard. cadvisor works on this rootless-Podman / cgroup-v2 host on one condition: --cgroupns=host. Podman starts every container in a private cgroup namespace, so without it cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the rootless app containers (it resolved 2 of 34), so naming comes from elsewhere: the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into a `container` label. No podman socket needed. cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 and the host LAN address — never a wildcard. Prometheus (a bridge container) reaches a host-published port at the LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports for these containers are graphed — CPU usage, memory working-set/limit, OOM — not network or filesystem, which are absent under the host-PID layout. Wired through the deployer (allowlist, deploy_observability case + /healthz gate, dispatcher), the planner (both targets), and observability_quadlet.bats.
This commit is contained in:
1 parent
961713e92d
commit
31908c623f
8 files changed
+398
-15
No files matched your search
@@ -0,0 +1,39 @@
|
||||
# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKSERVER.
|
||||
#
|
||||
# Install path on dankserver: ~/.config/containers/systemd/cadvisor.container
|
||||
# Unit name: cadvisor.service (container name: cadvisor)
|
||||
#
|
||||
# The Dankserver twin of dankclaw/observability/quadlet/cadvisor.container.
|
||||
# Same rootless shape, same --cgroupns=host requirement (see that file for the
|
||||
# full story), only the LAN address differs: this host publishes on
|
||||
# 192.168.0.235:8080 so Prometheus on Dankclaw can scrape it over the real
|
||||
# network (cross-host, so the host.containers.internal trick does not apply).
|
||||
# It reports the media/library stack (romm, plex, jellyfin, copyparty, calibre,
|
||||
# tautulli) by their quadlet unit names via the cgroup path.
|
||||
|
||||
[Unit]
|
||||
Description=cAdvisor container metrics (Dankserver)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=cadvisor
|
||||
Image=gcr.io/cadvisor/cadvisor:v0.49.1
|
||||
PodmanArgs=--pid=host --cgroupns=host
|
||||
PublishPort=127.0.0.1:8080:8080
|
||||
PublishPort=192.168.0.235:8080:8080
|
||||
Volume=/sys:/sys:ro
|
||||
Volume=/:/rootfs:ro
|
||||
Exec=--housekeeping_interval=30s
|
||||
HealthCmd=wget --spider -q http://localhost:8080/healthz
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=40s
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
Reference in new issue
Block a user