feat(observability): per-container metrics via cadvisor + Containers dashboard
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a `containers` dashboard. cadvisor works on this rootless-Podman / cgroup-v2 host on one condition: --cgroupns=host. Podman starts every container in a private cgroup namespace, so without it cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the rootless app containers (it resolved 2 of 34), so naming comes from elsewhere: the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into a `container` label. No podman socket needed. cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 and the host LAN address — never a wildcard. Prometheus (a bridge container) reaches a host-published port at the LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports for these containers are graphed — CPU usage, memory working-set/limit, OOM — not network or filesystem, which are absent under the host-PID layout. Wired through the deployer (allowlist, deploy_observability case + /healthz gate, dispatcher), the planner (both targets), and observability_quadlet.bats.
This commit is contained in:
@@ -10,9 +10,9 @@ timestamp: 2026-10-20
|
||||
|
||||
## Summary
|
||||
|
||||
Six containers and one host-side process: Prometheus (metrics), Grafana
|
||||
Seven containers and one host-side process: Prometheus (metrics), Grafana
|
||||
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
|
||||
metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
|
||||
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
|
||||
carrying every firing group to Signal. All of it is Quadlet-managed,
|
||||
boot-persistent and loopback-only, deployed by
|
||||
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
|
||||
@@ -34,6 +34,7 @@ that proves it today.
|
||||
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
|
||||
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
|
||||
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
|
||||
| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
|
||||
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
|
||||
|
||||
Every unit lives in `dankclaw/observability/quadlet/`, carries
|
||||
@@ -44,9 +45,10 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
|
||||
the same day, and `homepage-status.timer` — the root unit that probes all of
|
||||
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
|
||||
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
|
||||
Dankserver runs one member of this family, node-exporter, as its own unit
|
||||
(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
|
||||
`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
|
||||
Dankserver runs two members of this family, node-exporter and cadvisor, as
|
||||
its own units (`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
|
||||
`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
|
||||
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`).
|
||||
|
||||
## Exposure
|
||||
|
||||
@@ -98,10 +100,23 @@ verify it through the APIs.
|
||||
relay, and `journalctl --user -u signal-relay` records successful
|
||||
`sent to +12062476336:` lines for every rule in the set.
|
||||
* **Dashboards: working.** Provisioned datasources — Prometheus (uid
|
||||
`prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
|
||||
`prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
|
||||
`containers`, `observability` and `logs-overview` dashboards, all
|
||||
installed from `shared/grafana/provisioning/`. The Loki datasource is how
|
||||
you read logs: Loki has no web UI of its own, so Grafana Explore is the
|
||||
human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
|
||||
* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
|
||||
after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
|
||||
per-container CPU/memory here, and on this rootless-Podman host it works on one
|
||||
condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
|
||||
namespace, so without that flag cadvisor sees only its own cgroup root and
|
||||
exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
|
||||
names nothing itself — the libpod socket resolved only 2 of 34 rootless
|
||||
containers — but the quadlet unit name is carried in the cgroup path
|
||||
(`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
|
||||
`container` label. Network and filesystem are NOT exported for these containers
|
||||
(no netns/classid attribution under the host-PID layout), so the dashboard
|
||||
graphs only CPU, memory and OOM and does not fake the rest.
|
||||
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
|
||||
a rootless Alloy cannot read the `0640` journal files, and the container-log
|
||||
globs point at a root-storage layout that does not exist here. The fix is built
|
||||
@@ -117,7 +132,7 @@ verify it through the APIs.
|
||||
|
||||
## Citations
|
||||
|
||||
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
|
||||
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
|
||||
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
|
||||
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
|
||||
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
|
||||
|
||||
Reference in New Issue
Block a user