feat(observability): per-container metrics via cadvisor + Containers dashboard

Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.

cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.

cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.

Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
This commit is contained in:
Frank
2026-10-20 22:51:42 -07:00
parent 961713e92d
commit 31908c623f
8 changed files with 398 additions and 15 deletions
@@ -10,9 +10,9 @@ timestamp: 2026-10-20
## Summary
Six containers and one host-side process: Prometheus (metrics), Grafana
Seven containers and one host-side process: Prometheus (metrics), Grafana
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
carrying every firing group to Signal. All of it is Quadlet-managed,
boot-persistent and loopback-only, deployed by
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
@@ -34,6 +34,7 @@ that proves it today.
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
Every unit lives in `dankclaw/observability/quadlet/`, carries
@@ -44,9 +45,10 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
the same day, and `homepage-status.timer` — the root unit that probes all of
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
Dankserver runs one member of this family, node-exporter, as its own unit
(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
Dankserver runs two members of this family, node-exporter and cadvisor, as
its own units (`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`).
## Exposure
@@ -98,10 +100,23 @@ verify it through the APIs.
relay, and `journalctl --user -u signal-relay` records successful
`sent to +12062476336:` lines for every rule in the set.
* **Dashboards: working.** Provisioned datasources — Prometheus (uid
`prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
`prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
`containers`, `observability` and `logs-overview` dashboards, all
installed from `shared/grafana/provisioning/`. The Loki datasource is how
you read logs: Loki has no web UI of its own, so Grafana Explore is the
human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
per-container CPU/memory here, and on this rootless-Podman host it works on one
condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
namespace, so without that flag cadvisor sees only its own cgroup root and
exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
names nothing itself — the libpod socket resolved only 2 of 34 rootless
containers — but the quadlet unit name is carried in the cgroup path
(`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
`container` label. Network and filesystem are NOT exported for these containers
(no netns/classid attribution under the host-PID layout), so the dashboard
graphs only CPU, memory and OOM and does not fake the rest.
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
a rootless Alloy cannot read the `0640` journal files, and the container-log
globs point at a root-storage layout that does not exist here. The fix is built
@@ -117,7 +132,7 @@ verify it through the APIs.
## Citations
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`