dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(observability): per-container metrics via cadvisor + Containers dashboard

Browse Source 
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.

cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.

cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.

Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
This commit is contained in:
Frank
2026-10-20 22:51:42 -07:00
parent 961713e92d
commit 31908c623f
8 changed files with 398 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/observability_quadlet.bats
+72 -2
View File
@@ -21,7 +21,7 @@ load helpers
UNIT_DIR="$REPO_ROOT/dankclaw/observability/quadlet"
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
UNITS=(prometheus grafana loki alloy node-exporter alertmanager)
UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager)
# container_section <unit> — the [Container] key/value block only
container_section() {
@@ -33,7 +33,7 @@ service_section() {
awk '/^\[Service\]/{f=1;next} /^\[/{f=0} f' "$1"
}
@test "all six observability units exist under dankclaw/observability/quadlet/" {
@test "all seven observability units exist under dankclaw/observability/quadlet/" {
local svc
for svc in "${UNITS[@]}"; do
[ -f "$UNIT_DIR/$svc.container" ] || { echo "missing $svc.container"; return 1; }
@@ -277,6 +277,7 @@ loki http://127.0.0.1:3100/ready
alloy http://127.0.0.1:12345/-/healthy
node-exporter http://127.0.0.1:9100/metrics
alertmanager http://127.0.0.1:9093/-/healthy
cadvisor http://127.0.0.1:8080/healthz
PAIRS
}
@@ -568,3 +569,72 @@ assert any('ALERTS' in e for e in exprs), exprs
[ "$status" -eq 0 ]
[ "$output" = "alertmanager" ]
}
@test "cadvisor exports container metrics only because of --cgroupns=host, and binds exactly its two consumers" {
# Podman starts every container in a private cgroup namespace, so a cadvisor
# without --cgroupns=host sees only its own cgroup root and exports a single
# series (measured 2026-10-20: 1 series before the flag, 996 after). That
# flag and the host /sys mount are the entire mechanism — pin them, plus the
# publish set: loopback (host probe + health gate) and this host's LAN
# address (Prometheus reaches a host-published port there). Never a wildcard.
local unit lan
for pair in "dankclaw/observability/quadlet:192.168.0.143" "dankserver/observability/quadlet:192.168.0.235"; do
unit="$REPO_ROOT/${pair%%:*}/cadvisor.container"
lan="${pair##*:}"
[ -f "$unit" ] || { echo "missing $unit"; return 1; }
grep -q -- '--cgroupns=host' "$unit" || { echo "$unit lacks --cgroupns=host"; return 1; }
grep -q -- '--pid=host' "$unit" || { echo "$unit lacks --pid=host"; return 1; }
grep -q '^Volume=/sys:/sys:ro$' "$unit" || { echo "$unit lacks the host /sys mount"; return 1; }
grep -q '^WantedBy=default.target$' "$unit" || { echo "$unit is not boot-persistent"; return 1; }
grep -q '^Restart=always$' "$unit" || { echo "$unit is not self-healing"; return 1; }
grep -q '^HealthCmd=' "$unit" || { echo "$unit has no health check"; return 1; }
grep -q "^PublishPort=127.0.0.1:8080:8080\$" "$unit" || { echo "$unit must publish loopback :8080"; return 1; }
grep -q "^PublishPort=$lan:8080:8080\$" "$unit" || { echo "$unit must publish $lan:8080"; return 1; }
! grep -qE '^PublishPort=(0\.0\.0\.0|:)' "$unit" || { echo "$unit publishes on a wildcard"; return 1; }
done
}
@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
# Cross-file contract: cadvisor publishes the LAN port; a cadvisor scrape
# job pointing anywhere else reads nothing (unlike node-exporter's host-net
# bind, a published port IS reachable from this bridge container at the LAN
# address). And the whole dashboard depends on the id -> container relabel:
# cadvisor names nothing itself, the quadlet unit name is only in the cgroup
# path. If either half changes alone, the Containers dashboard is blank.
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
grep -q '"192.168.0.143:8080"' "$cfg" || { echo "cadvisor-dankclaw does not scrape the published LAN port"; return 1; }
grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
# The relabel that turns …/app.slice/<name>.service… into a `container`.
grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }
grep -q 'target_label: container' "$cfg" || { echo "relabel does not set the container label"; return 1; }
}
@test "the Containers dashboard is provisioned from git" {
local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/containers.json"
[ -f "$dash" ]
python3 -c "
import json
d = json.load(open('$dash'))
assert d['uid'] == 'containers', d['uid']
names = [t['expr'] for p in d['panels'] for t in p.get('targets', [])]
# Only families cadvisor actually exports for rootless quadlet containers are
# graphed; network/fs are absent for them, so a network/fs panel would be a
# permanent empty lie.
assert any('container_cpu_usage_seconds_total' in e for e in names), names
assert any('container_memory_working_set_bytes' in e for e in names), names
assert not any('container_network' in e or 'container_fs_' in e for e in names), names
"
}
@test "the planner selects cadvisor for its unit on each target" {
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
--paths dankclaw/observability/quadlet/cadvisor.container
[ "$status" -eq 0 ]
[ "$output" = "cadvisor" ]
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver \
--paths dankserver/observability/quadlet/cadvisor.container
[ "$status" -eq 0 ]
[ "$output" = "cadvisor" ]
}