dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(observability): per-container metrics via cadvisor + Containers dashboard

Browse Source 
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.

cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.

cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.

Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
This commit is contained in:
Frank
2026-10-20 22:51:42 -07:00
parent 961713e92d
commit 31908c623f
8 changed files with 398 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/deploy-remote.sh
+6 -3
View File
@@ -86,7 +86,7 @@ service_is_allowed() {
dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
dankclaw:comfyui|dankclaw:hermes|\
dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:alertmanager|\
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:alertmanager|\
dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -97,7 +97,7 @@ service_is_allowed() {
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
dankserver:frank-archon|dankserver:node-exporter|\
dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|\
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
frankbook-pro:readarr)
@@ -520,6 +520,8 @@ deploy_observability() {
;;
node-exporter)
;;
cadvisor)
;;
*)
echo "unknown observability service: $service" >&2
return 2
@@ -533,6 +535,7 @@ deploy_observability() {
grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
esac
}
@@ -2000,7 +2003,7 @@ deploy_service() {
plex) deploy_plex ;;
tautulli) deploy_tautulli ;;
alloy) deploy_alloy ;;
prometheus|grafana|loki|node-exporter|alertmanager) deploy_observability "$service" ;;
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
comfyui) deploy_comfyui ;;
*) deploy_compose_service "$service" ;;
esac