Files

T

Frank 31908c623f feat(observability): per-container metrics via cadvisor + Containers dashboard

Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.

cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.

cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.

Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.

2026-10-20 22:51:42 -07:00

9.0 KiB

Raw Permalink Blame History

type, title, description, tags, timestamp

type

title

description

Observability on Dankclaw

Summary

Seven containers and one host-side process: Prometheus (metrics), Grafana (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), with signal-relay on the host carrying every firing group to Signal. All of it is Quadlet-managed, boot-persistent and loopback-only, deployed by scripts/deploy-remote.sh → deploy_observability. The metrics half works end to end; the log half is configured, healthy, and carries no data.

This body was rewritten on 2026-10-18 from live evidence: the previous version was deleted down to its front matter by commit 93d1181 (the "OKF v0.1 compliance pass"), while sibling pages kept pointing at a "state note" on this page that no longer existed. Every statement below carries the command or file that proves it today.

The stack

Service	Container	Published	Unit	Page
Prometheus	`docker.io/prom/prometheus:latest`	`127.0.0.1:9090`	`prometheus.container`	Prometheus
Alertmanager	`docker.io/prom/alertmanager:latest`	`127.0.0.1:9093`	`alertmanager.container`	Alertmanager
Grafana	`docker.io/grafana/grafana:latest`	`127.0.0.1:3002` → 3000	`grafana.container`	Grafana
Loki	`localhost/franksdank/loki:latest`	`127.0.0.1:3100`	`loki.container`	Loki
Alloy	`localhost/franksdank/alloy:latest`	`127.0.0.1:12345`	`alloy.container`	Alloy
node-exporter	`docker.io/prom/node-exporter:latest`	host netns: `127.0.0.1:9100` + `192.168.0.143:9100`	`node-exporter.container`	Node-Exporter
cadvisor	`gcr.io/cadvisor/cadvisor:v0.49.1`	bridge publish `127.0.0.1:8080` + `192.168.0.143:8080`	`cadvisor.container`	—
Signal relay	host `python3`, not a container	`192.168.0.143:18093`	`scripts/signal-relay.service` (user unit)	Alertmanager

Every unit lives in dankclaw/observability/quadlet/, carries Restart=always/RestartSec=10 and WantedBy=default.target, is installed at ~/.config/containers/systemd/<name>.container, and reports generated + active (systemctl --user is-enabled|is-active for all six, 2026-10-18; linger is on for dank). signal-relay.service reports enabled + active the same day, and homepage-status.timer — the root unit that probes all of these on loopback — is enabled + active (deploy-remote.sh:319-321 copies the probe registry shared/homepage/services.json to /srv/homepage). Dankserver runs two members of this family, node-exporter and cadvisor, as its own units (dankserver/observability/quadlet/{node-exporter,cadvisor}.container, generated + active; node-exporter binds 127.0.0.1:9100 + 192.168.0.235:9100, cadvisor publishes 127.0.0.1:8080 + 192.168.0.235:8080).

Exposure

The five HTTP surfaces publish on 127.0.0.1 only — ss -ltnp (2026-10-18) shows 127.0.0.1:9090, 127.0.0.1:9093, 127.0.0.1:3002, 127.0.0.1:3100 127.0.0.1:12345 and no wildcard — because each of them is unauthenticated inside, and several carry lifecycle or query APIs. Browser access goes through Caddy: each vhost (dankclaw/Caddyfile:128-145) proxies to the guard anubis:8923, and the post-auth handlers route by container name on dankclaw_default — loki.franksdank.cloud → loki:3100 (:364-366) grafana.franksdank.cloud → grafana:3000 (:374-376) prometheus.franksdank.cloud → prometheus:9090 (:379-381). Alertmanager has no vhost at all. node-exporter is the exception that must live in the host network namespace, and it binds exactly two addresses rather than the wildcard — see Node-Exporter.

The relay's bind is the one address a bridge container can reach on the host: 192.168.0.143:18093, with iptables INPUT admitting loopback, the host itself and the podman bridge subnet 10.89.0.0/16 before the DROP (scripts/deploy-remote.sh:255-262).

Deployment

./scripts/deploy.sh dankclaw --service <name> runs deploy_observability which installs the unit, copies that service's config (copy_prometheus_config, copy_alertmanager_config, copy_loki_config copy_alloy_config, copy_grafana_provisioning, plus install_signal_relay with the alertmanager deploy), then waits on the loopback health endpoint (/prometheus: /-/healthy, /alertmanager: /-/healthy /grafana: /api/health, /loki: /ready, /alloy: /-/healthy /node-exporter: /metrics). Startup order follows the data dependencies — grafana after prometheus, alloy after loki (grafana.container:32-33, alloy.container:38-39). See Observability Provisioning for what is provisioned as code and how to verify it through the APIs.

Current state (2026-10-20)

Metrics: working. curl -s http://127.0.0.1:9090/api/v1/targets reports eight up targets — the three LLM endpoints (frank-archon on vllm:*, frank-dragoon and frank-zealot on llamacpp:*), grafana, loki, prometheus and both node exporters. The four downs are structural (the comfyui and services HTML probes, and Gitea's disabled metrics endpoint), not outages. Two rule groups in shared/prometheus/rules/llm-fleet.yml: llm-fleet (2 recording + 4 alert, llamacpp:*) and vllm-fleet (1 recording + 4 alert, vllm:* — archon queueing, prefix collapse, preemption, p95 time-to-first-token), each keyed only to the namespace its runtime exports.
Delivery: working. Alertmanager's single signal receiver webhooks the relay, and journalctl --user -u signal-relay records successful sent to +12062476336: lines for every rule in the set.
Dashboards: working. Provisioned datasources — Prometheus (uid prometheus) and Loki (uid loki) — and the llm-fleet, host-system, containers, observability and logs-overview dashboards, all installed from shared/grafana/provisioning/. The Loki datasource is how you read logs: Loki has no web UI of its own, so Grafana Explore is the human surface (the guarded loki.franksdank.cloud vhost serves only the API).
Per-container metrics: the containers dashboard (added 2026-10-20; live after deploy.sh <host> --service cadvisor). cadvisor is the only way to see per-container CPU/memory here, and on this rootless-Podman host it works on one condition: --cgroupns=host. Podman starts every container in a PRIVATE cgroup namespace, so without that flag cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor names nothing itself — the libpod socket resolved only 2 of 34 rootless containers — but the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/…), so prometheus.yml relabels that path into a container label. Network and filesystem are NOT exported for these containers (no netns/classid attribution under the host-PID layout), so the dashboard graphs only CPU, memory and OOM and does not fake the rest.
Logs: fixed in code, awaiting deploy. The pipeline shipped nothing because a rootless Alloy cannot read the 0640 journal files, and the container-log globs point at a root-storage layout that does not exist here. The fix is built and tested in-repo (2026-10-20): Alloy is now a root system service on Network=host, reading system.journal + user-*.journal and pushing to Loki's 127.0.0.1:3100 loopback publish — a throwaway root probe read 31,825 lines and pushed 26,966 with zero drops. The live instance is still the old rootless unit until deploy.sh dankclaw --service alloy runs. Detail in Alloy → Why root; Loki's store state in Loki → Contents.
Not configured: no blackbox exporter (the structural downs have no real uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but not yet deployed (see Logs above).

Citations

dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container
scripts/deploy-remote.sh:219-300 (copiers + relay), :521-536 (health waits), :477-486 (grafana env seed)
scripts/signal-relay.py:26-49, scripts/signal-relay.service
shared/prometheus/prometheus.yml, shared/prometheus/alertmanager.yml, shared/prometheus/rules/llm-fleet.yml
shared/grafana/provisioning/, shared/loki/loki-config.yaml, shared/alloy/config.alloy
tests/observability_quadlet.bats (boot persistence, no wildcard publish, probe addresses, delivery contract)
Host evidence 2026-10-18: ss -ltnp, podman ps, systemctl --user is-enabled|is-active, curl on 127.0.0.1:9090/api/v1/{targets,rules,alerts}, 127.0.0.1:3100/loki/api/v1/labels, 127.0.0.1:12345/metrics, 127.0.0.1:3002/api/health, 192.168.0.143:18093/health, journalctl --user -u signal-relay

9.0 KiB Raw Permalink Blame History