Per-container CPU/memory for the quadlet app stacks was invisible — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a containers dashboard, wired through the deployer, the planner, and the observability test seam.
Why it works on this host
cadvisor on rootless Podman / cgroup-v2 fails silently by default. It works only with --cgroupns=host: Podman starts every container in a private cgroup namespace, so a containerized cadvisor sees only its own cgroup root — 1 series — until the flag is added, then 996 per-container series (measured 2026-10-20).
Naming does not come from cadvisor's libpod socket (it resolved only 2 of 34 rootless app containers). It comes from the cgroup path, which carries the quadlet unit name — app.slice/<name>.service/libpod-payload-<hex> — and prometheus.ymlrelabels that path into a container label. No podman socket needed.
cadvisor can't bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 + the host LAN address — never a wildcard. Prometheus reaches a host-published port at the LAN address, so the jobs scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports are graphed — CPU, memory working-set/limit, OOM — not network/filesystem (absent under the host-PID layout). No empty panels.
After merge: deploy.sh dankclaw --service cadvisor then … dankserver --service cadvisor; /healthz answers, both cadvisor-* targets up, the containers dashboard is non-empty
Built and tested in the /tmp/wt-containers worktree (branch feat/containers-observability).
## What
Per-container CPU/memory for the quadlet app stacks was invisible — Grafana could only see the host (`node_exporter`) and the LLM engines. This adds a **rootless `cadvisor` quadlet on each host** and a **`containers`** dashboard, wired through the deployer, the planner, and the observability test seam.
## Why it works on this host
cadvisor on rootless Podman / cgroup-v2 fails silently by default. It works only with **`--cgroupns=host`**: Podman starts every container in a *private* cgroup namespace, so a containerized cadvisor sees only its own cgroup root — **1 series** — until the flag is added, then **996** per-container series (measured 2026-10-20).
Naming does **not** come from cadvisor's libpod socket (it resolved only **2 of 34** rootless app containers). It comes from the cgroup *path*, which carries the quadlet unit name — `app.slice/<name>.service/libpod-payload-<hex>` — and `prometheus.yml` **relabels** that path into a `container` label. No podman socket needed.
cadvisor can't bind per-interface like `node_exporter`, so each unit is a bridge container publishing exactly `127.0.0.1:8080` + the host LAN address — never a wildcard. Prometheus reaches a host-**published** port at the LAN address, so the jobs scrape `192.168.0.143:8080` / `192.168.0.235:8080`. Only the families cadvisor actually exports are graphed — CPU, memory working-set/limit, OOM — **not** network/filesystem (absent under the host-PID layout). No empty panels.
## Files
- `dankclaw/observability/quadlet/cadvisor.container`, `dankserver/…/cadvisor.container`
- `shared/prometheus/prometheus.yml` — `cadvisor-*` jobs + id→`container` relabel
- `shared/grafana/provisioning/dashboards/containers.json`
- `scripts/deploy-remote.sh`, `scripts/deploy-plan.py` — allowlist, observability case + `/healthz` gate, dispatcher, planner (both targets)
- `tests/observability_quadlet.bats` — cadvisor into `UNITS` + three new tests
- `serverAppConfigs-docs/Infrastructure/Observability.md` — roster + the cgroupns law
## Test plan
- [x] `./tests/run.sh` — 476 pass (incl. vault link-integrity, OKF, cadvisor seams)
- [ ] After merge: `deploy.sh dankclaw --service cadvisor` then `… dankserver --service cadvisor`; `/healthz` answers, both `cadvisor-*` targets up, the `containers` dashboard is non-empty
> Built and tested in the `/tmp/wt-containers` worktree (branch `feat/containers-observability`).
---
cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 6d1fc1f
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.
cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.
cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.
Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
dank
merged commit 31b71a16d6 into main2026-10-20 23:08:24 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
Per-container CPU/memory for the quadlet app stacks was invisible — Grafana could only see the host (
node_exporter) and the LLM engines. This adds a rootlesscadvisorquadlet on each host and acontainersdashboard, wired through the deployer, the planner, and the observability test seam.Why it works on this host
cadvisor on rootless Podman / cgroup-v2 fails silently by default. It works only with
--cgroupns=host: Podman starts every container in a private cgroup namespace, so a containerized cadvisor sees only its own cgroup root — 1 series — until the flag is added, then 996 per-container series (measured 2026-10-20).Naming does not come from cadvisor's libpod socket (it resolved only 2 of 34 rootless app containers). It comes from the cgroup path, which carries the quadlet unit name —
app.slice/<name>.service/libpod-payload-<hex>— andprometheus.ymlrelabels that path into acontainerlabel. No podman socket needed.cadvisor can't bind per-interface like
node_exporter, so each unit is a bridge container publishing exactly127.0.0.1:8080+ the host LAN address — never a wildcard. Prometheus reaches a host-published port at the LAN address, so the jobs scrape192.168.0.143:8080/192.168.0.235:8080. Only the families cadvisor actually exports are graphed — CPU, memory working-set/limit, OOM — not network/filesystem (absent under the host-PID layout). No empty panels.Files
dankclaw/observability/quadlet/cadvisor.container,dankserver/…/cadvisor.containershared/prometheus/prometheus.yml—cadvisor-*jobs + id→containerrelabelshared/grafana/provisioning/dashboards/containers.jsonscripts/deploy-remote.sh,scripts/deploy-plan.py— allowlist, observability case +/healthzgate, dispatcher, planner (both targets)tests/observability_quadlet.bats— cadvisor intoUNITS+ three new testsserverAppConfigs-docs/Infrastructure/Observability.md— roster + the cgroupns lawTest plan
./tests/run.sh— 476 pass (incl. vault link-integrity, OKF, cadvisor seams)deploy.sh dankclaw --service cadvisorthen… dankserver --service cadvisor;/healthzanswers, bothcadvisor-*targets up, thecontainersdashboard is non-emptycwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 6d1fc1f