Frank
31908c623f
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a `containers` dashboard. cadvisor works on this rootless-Podman / cgroup-v2 host on one condition: --cgroupns=host. Podman starts every container in a private cgroup namespace, so without it cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the rootless app containers (it resolved 2 of 34), so naming comes from elsewhere: the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into a `container` label. No podman socket needed. cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 and the host LAN address — never a wildcard. Prometheus (a bridge container) reaches a host-published port at the LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports for these containers are graphed — CPU usage, memory working-set/limit, OOM — not network or filesystem, which are absent under the host-PID layout. Wired through the deployer (allowlist, deploy_observability case + /healthz gate, dispatcher), the planner (both targets), and observability_quadlet.bats.
142 lines
9.0 KiB
Markdown
142 lines
9.0 KiB
Markdown
---
|
|
type: reference
|
|
title: "Observability on Dankclaw"
|
|
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
|
|
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
|
|
timestamp: 2026-10-20
|
|
---
|
|
|
|
# Observability on Dankclaw
|
|
|
|
## Summary
|
|
|
|
Seven containers and one host-side process: Prometheus (metrics), Grafana
|
|
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
|
|
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
|
|
carrying every firing group to Signal. All of it is Quadlet-managed,
|
|
boot-persistent and loopback-only, deployed by
|
|
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
|
|
end to end; the log half is configured, healthy, and carries no data.
|
|
|
|
This body was rewritten on 2026-10-18 from live evidence: the previous version
|
|
was deleted down to its front matter by commit `93d1181` (the "OKF v0.1
|
|
compliance pass"), while sibling pages kept pointing at a "state note" on this
|
|
page that no longer existed. Every statement below carries the command or file
|
|
that proves it today.
|
|
|
|
## The stack
|
|
|
|
| Service | Container | Published | Unit | Page |
|
|
|---|---|---|---|---|
|
|
| Prometheus | `docker.io/prom/prometheus:latest` | `127.0.0.1:9090` | `prometheus.container` | [[Prometheus]] |
|
|
| Alertmanager | `docker.io/prom/alertmanager:latest` | `127.0.0.1:9093` | `alertmanager.container` | [[Alertmanager]] |
|
|
| Grafana | `docker.io/grafana/grafana:latest` | `127.0.0.1:3002` → 3000 | `grafana.container` | [[Grafana]] |
|
|
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
|
|
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
|
|
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
|
|
| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
|
|
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
|
|
|
|
Every unit lives in `dankclaw/observability/quadlet/`, carries
|
|
`Restart=always`/`RestartSec=10` and `WantedBy=default.target`, is installed at
|
|
`~/.config/containers/systemd/<name>.container`, and reports `generated` +
|
|
`active` (`systemctl --user is-enabled|is-active` for all six, 2026-10-18;
|
|
linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
|
|
the same day, and `homepage-status.timer` — the root unit that probes all of
|
|
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
|
|
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
|
|
Dankserver runs two members of this family, node-exporter and cadvisor, as
|
|
its own units (`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
|
|
`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
|
|
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`).
|
|
|
|
## Exposure
|
|
|
|
The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
|
|
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`
|
|
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
|
|
inside, and several carry lifecycle or query APIs. Browser access goes through
|
|
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
|
|
`anubis:8923`, and the post-auth handlers route by container name on
|
|
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`)
|
|
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`)
|
|
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
|
|
Alertmanager has no vhost at all. node-exporter is the exception that must live
|
|
in the host network namespace, and it binds exactly two addresses rather than
|
|
the wildcard — see [[Node-Exporter]].
|
|
|
|
The relay's bind is the one address a bridge container can reach on the host:
|
|
`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
|
|
and the podman bridge subnet `10.89.0.0/16` before the DROP
|
|
(`scripts/deploy-remote.sh:255-262`).
|
|
|
|
## Deployment
|
|
|
|
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`
|
|
which installs the unit, copies that service's config
|
|
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`
|
|
`copy_alloy_config`, `copy_grafana_provisioning`, plus
|
|
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
|
|
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
|
|
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`
|
|
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
|
|
`grafana` after `prometheus`, `alloy` after `loki`
|
|
(`grafana.container:32-33`, `alloy.container:38-39`). See
|
|
[[Observability Provisioning]] for what is provisioned as code and how to
|
|
verify it through the APIs.
|
|
|
|
## Current state (2026-10-20)
|
|
|
|
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
|
|
eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
|
|
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
|
|
and both node exporters. The four downs are structural (the `comfyui` and
|
|
`services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
|
|
Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
|
|
(2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
|
|
alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
|
|
time-to-first-token), each keyed only to the namespace its runtime exports.
|
|
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
|
|
relay, and `journalctl --user -u signal-relay` records successful
|
|
`sent to +12062476336:` lines for every rule in the set.
|
|
* **Dashboards: working.** Provisioned datasources — Prometheus (uid
|
|
`prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
|
|
`containers`, `observability` and `logs-overview` dashboards, all
|
|
installed from `shared/grafana/provisioning/`. The Loki datasource is how
|
|
you read logs: Loki has no web UI of its own, so Grafana Explore is the
|
|
human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
|
|
* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
|
|
after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
|
|
per-container CPU/memory here, and on this rootless-Podman host it works on one
|
|
condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
|
|
namespace, so without that flag cadvisor sees only its own cgroup root and
|
|
exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
|
|
names nothing itself — the libpod socket resolved only 2 of 34 rootless
|
|
containers — but the quadlet unit name is carried in the cgroup path
|
|
(`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
|
|
`container` label. Network and filesystem are NOT exported for these containers
|
|
(no netns/classid attribution under the host-PID layout), so the dashboard
|
|
graphs only CPU, memory and OOM and does not fake the rest.
|
|
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
|
|
a rootless Alloy cannot read the `0640` journal files, and the container-log
|
|
globs point at a root-storage layout that does not exist here. The fix is built
|
|
and tested in-repo (2026-10-20): Alloy is now a **root system service on
|
|
`Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
|
|
Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
|
|
lines and pushed 26,966 with zero drops. The **live** instance is still the old
|
|
rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
|
|
[[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
|
|
* **Not configured:** no blackbox exporter (the structural downs have no real
|
|
uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
|
|
not yet deployed (see *Logs* above).
|
|
|
|
## Citations
|
|
|
|
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
|
|
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
|
|
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
|
|
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
|
|
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
|
|
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
|
|
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
|