Frank
c41384e03d
copy_grafana_provisioning named llm-fleet.json alone, so a new board in shared/grafana/provisioning/dashboards/ silently never reached /opt/grafana/provisioning. Measured 2026-10-21: the containers board was merged, seam-tested, and documented as working since 2026-10-20, yet was NEVER provisioned — its panel fixes were undeployable; the other four boards existed on disk only because someone hand-copied them once (and host-system.json there had gone stale against the repo). The copier now globs the directory, and GRAFANA_PROV_ROOT lets the new bats case run the copier for real into a sandbox (red against the old hand-listed body, green against the glob). Vault: the copier law + the containers incident on Observability Provisioning, the corrected dashboards claim on Observability. Note: a mktemp sandbox + trap RETURN races bats' trap machinery (the dir vanishes mid-function); BATS_TEST_TMPDIR is the safe idiom.
168 lines
11 KiB
Markdown
168 lines
11 KiB
Markdown
---
|
|
type: reference
|
|
title: "Observability on Dankclaw"
|
|
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
|
|
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
|
|
timestamp: 2026-10-20
|
|
---
|
|
|
|
# Observability on Dankclaw
|
|
|
|
## Summary
|
|
|
|
Seven containers and two host-side processes: Prometheus (metrics), Grafana
|
|
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
|
|
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
|
|
plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
|
|
boot-persistent and loopback-only, deployed by
|
|
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
|
|
end to end; the log half is configured, healthy, and carries no data.
|
|
|
|
This body was rewritten on 2026-10-18 from live evidence: the previous version
|
|
was deleted down to its front matter by commit `93d1181` (the "OKF v0.1
|
|
compliance pass"), while sibling pages kept pointing at a "state note" on this
|
|
page that no longer existed. Every statement below carries the command or file
|
|
that proves it today.
|
|
|
|
## The stack
|
|
|
|
| Service | Container | Published | Unit | Page |
|
|
|---|---|---|---|---|
|
|
| Prometheus | `docker.io/prom/prometheus:latest` | `127.0.0.1:9090` | `prometheus.container` | [[Prometheus]] |
|
|
| Alertmanager | `docker.io/prom/alertmanager:latest` | `127.0.0.1:9093` | `alertmanager.container` | [[Alertmanager]] |
|
|
| Grafana | `docker.io/grafana/grafana:latest` | `127.0.0.1:3002` → 3000 | `grafana.container` | [[Grafana]] |
|
|
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
|
|
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
|
|
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
|
|
| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
|
|
| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
|
|
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
|
|
|
|
Every unit lives in `dankclaw/observability/quadlet/`, carries
|
|
`Restart=always`/`RestartSec=10` and `WantedBy=default.target`, is installed at
|
|
`~/.config/containers/systemd/<name>.container`, and reports `generated` +
|
|
`active` (`systemctl --user is-enabled|is-active` for all six, 2026-10-18;
|
|
linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
|
|
the same day, and `homepage-status.timer` — the root unit that probes all of
|
|
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
|
|
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
|
|
Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
|
|
(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
|
|
`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
|
|
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
|
|
`gpu-exporter` (a user unit installed the same way as `signal-relay`).
|
|
|
|
## Exposure
|
|
|
|
The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
|
|
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`
|
|
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
|
|
inside, and several carry lifecycle or query APIs. Browser access goes through
|
|
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
|
|
`anubis:8923`, and the post-auth handlers route by container name on
|
|
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`)
|
|
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`)
|
|
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
|
|
Alertmanager has no vhost at all. node-exporter is the exception that must live
|
|
in the host network namespace, and it binds exactly two addresses rather than
|
|
the wildcard — see [[Node-Exporter]].
|
|
|
|
**Same-host reachability from a bridge container is measured, not inferred**
|
|
(2026-10-21, from inside the Prometheus container): `host.containers.internal`
|
|
(the podman gateway) reaches host services whether they bind host-net or are
|
|
published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
|
|
— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
|
|
every same-host Prometheus target uses the gateway and only cross-host targets
|
|
(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
|
|
relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
|
|
iptables INPUT admitting loopback, the host itself and the podman bridge subnet
|
|
`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
|
|
point a scrape target at a host LAN IP from its own host on the strength of
|
|
one working example; dial the gateway and probe.
|
|
|
|
## Deployment
|
|
|
|
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`
|
|
which installs the unit, copies that service's config
|
|
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`
|
|
`copy_alloy_config`, `copy_grafana_provisioning`, plus
|
|
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
|
|
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
|
|
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`
|
|
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
|
|
`grafana` after `prometheus`, `alloy` after `loki`
|
|
(`grafana.container:32-33`, `alloy.container:38-39`). See
|
|
[[Observability Provisioning]] for what is provisioned as code and how to
|
|
verify it through the APIs.
|
|
|
|
## Current state (2026-10-20)
|
|
|
|
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
|
|
eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
|
|
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
|
|
and both node exporters. The four downs are structural (the `comfyui` and
|
|
`services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
|
|
Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
|
|
(2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
|
|
alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
|
|
time-to-first-token), each keyed only to the namespace its runtime exports.
|
|
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
|
|
relay, and `journalctl --user -u signal-relay` records successful
|
|
`sent to +12062476336:` lines for every rule in the set.
|
|
* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
|
|
datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
|
|
`llm-fleet`, `host-system`, `containers`, `observability` and
|
|
`logs-overview` boards. Until 2026-10-21 the copier carried only
|
|
`llm-fleet.json` by name and the `containers` board had never been
|
|
provisioned; the deployer now globs the directory. The Loki datasource is
|
|
how you read logs: Loki has no web UI of its own, so Grafana Explore is the
|
|
human surface (the guarded `loki.franksdank.cloud` vhost serves only the
|
|
API). Verify method and the copier law: [[Observability Provisioning]].
|
|
* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
|
|
after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
|
|
per-container CPU/memory here, and on this rootless-Podman host it works on one
|
|
condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
|
|
namespace, so without that flag cadvisor sees only its own cgroup root and
|
|
exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
|
|
names nothing itself — the libpod socket resolved only 2 of 34 rootless
|
|
containers — but the quadlet unit name is carried in the cgroup path
|
|
(`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
|
|
`container` label. Network and filesystem are NOT exported for these containers
|
|
(no netns/classid attribution under the host-PID layout), so the dashboard
|
|
graphs only CPU, memory and OOM and does not fake the rest.
|
|
* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
|
|
`deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
|
|
host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
|
|
own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
|
|
without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
|
|
`mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
|
|
emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
|
|
°C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
|
|
job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
|
|
`power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
|
|
the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
|
|
node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
|
|
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
|
|
a rootless Alloy cannot read the `0640` journal files, and the container-log
|
|
globs point at a root-storage layout that does not exist here. The fix is built
|
|
and tested in-repo (2026-10-20): Alloy is now a **root system service on
|
|
`Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
|
|
Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
|
|
lines and pushed 26,966 with zero drops. The **live** instance is still the old
|
|
rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
|
|
[[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
|
|
* **Not configured:** no blackbox exporter (the structural downs have no real
|
|
uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
|
|
not yet deployed (see *Logs* above).
|
|
|
|
## Citations
|
|
|
|
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
|
|
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
|
|
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
|
|
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
|
|
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
|
|
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
|
|
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
|
|
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
|