Frank
650b6e0281
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and cadvisor-dankclaw DOWN beside a green deploy: - gpu-exporter.py built one ThreadingHTTPServer per bind but ran serve_forever() on servers[0] only: the LAN-class socket sat listen()ing but never accept()ed, loopback deploy gate passed, every scrape that reached the other bind timed out on the SYN backlog. Now one thread per extra bind. Regression test spins --serve on two loopback binds and requires BOTH to answer /metrics. - cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the Prometheus container: a bridge container cannot dial its own host's LAN address at all (hairpin refuses published ports too); the podman gateway answers. Target moved to host.containers.internal:8080; the seam that pinned the LAN address (asserting the wrong doctrine) is re-pinned to the gateway plus a negative pin against the own-host LAN dial. Observability page records the measured reachability law replacing the inferred one.
165 lines
11 KiB
Markdown
165 lines
11 KiB
Markdown
---
|
|
type: reference
|
|
title: "Observability on Dankclaw"
|
|
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
|
|
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
|
|
timestamp: 2026-10-20
|
|
---
|
|
|
|
# Observability on Dankclaw
|
|
|
|
## Summary
|
|
|
|
Seven containers and two host-side processes: Prometheus (metrics), Grafana
|
|
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
|
|
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
|
|
plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
|
|
boot-persistent and loopback-only, deployed by
|
|
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
|
|
end to end; the log half is configured, healthy, and carries no data.
|
|
|
|
This body was rewritten on 2026-10-18 from live evidence: the previous version
|
|
was deleted down to its front matter by commit `93d1181` (the "OKF v0.1
|
|
compliance pass"), while sibling pages kept pointing at a "state note" on this
|
|
page that no longer existed. Every statement below carries the command or file
|
|
that proves it today.
|
|
|
|
## The stack
|
|
|
|
| Service | Container | Published | Unit | Page |
|
|
|---|---|---|---|---|
|
|
| Prometheus | `docker.io/prom/prometheus:latest` | `127.0.0.1:9090` | `prometheus.container` | [[Prometheus]] |
|
|
| Alertmanager | `docker.io/prom/alertmanager:latest` | `127.0.0.1:9093` | `alertmanager.container` | [[Alertmanager]] |
|
|
| Grafana | `docker.io/grafana/grafana:latest` | `127.0.0.1:3002` → 3000 | `grafana.container` | [[Grafana]] |
|
|
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
|
|
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
|
|
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
|
|
| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
|
|
| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
|
|
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
|
|
|
|
Every unit lives in `dankclaw/observability/quadlet/`, carries
|
|
`Restart=always`/`RestartSec=10` and `WantedBy=default.target`, is installed at
|
|
`~/.config/containers/systemd/<name>.container`, and reports `generated` +
|
|
`active` (`systemctl --user is-enabled|is-active` for all six, 2026-10-18;
|
|
linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
|
|
the same day, and `homepage-status.timer` — the root unit that probes all of
|
|
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
|
|
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
|
|
Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
|
|
(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
|
|
`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
|
|
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
|
|
`gpu-exporter` (a user unit installed the same way as `signal-relay`).
|
|
|
|
## Exposure
|
|
|
|
The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
|
|
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`
|
|
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
|
|
inside, and several carry lifecycle or query APIs. Browser access goes through
|
|
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
|
|
`anubis:8923`, and the post-auth handlers route by container name on
|
|
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`)
|
|
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`)
|
|
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
|
|
Alertmanager has no vhost at all. node-exporter is the exception that must live
|
|
in the host network namespace, and it binds exactly two addresses rather than
|
|
the wildcard — see [[Node-Exporter]].
|
|
|
|
**Same-host reachability from a bridge container is measured, not inferred**
|
|
(2026-10-21, from inside the Prometheus container): `host.containers.internal`
|
|
(the podman gateway) reaches host services whether they bind host-net or are
|
|
published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
|
|
— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
|
|
every same-host Prometheus target uses the gateway and only cross-host targets
|
|
(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
|
|
relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
|
|
iptables INPUT admitting loopback, the host itself and the podman bridge subnet
|
|
`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
|
|
point a scrape target at a host LAN IP from its own host on the strength of
|
|
one working example; dial the gateway and probe.
|
|
|
|
## Deployment
|
|
|
|
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`
|
|
which installs the unit, copies that service's config
|
|
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`
|
|
`copy_alloy_config`, `copy_grafana_provisioning`, plus
|
|
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
|
|
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
|
|
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`
|
|
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
|
|
`grafana` after `prometheus`, `alloy` after `loki`
|
|
(`grafana.container:32-33`, `alloy.container:38-39`). See
|
|
[[Observability Provisioning]] for what is provisioned as code and how to
|
|
verify it through the APIs.
|
|
|
|
## Current state (2026-10-20)
|
|
|
|
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
|
|
eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
|
|
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
|
|
and both node exporters. The four downs are structural (the `comfyui` and
|
|
`services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
|
|
Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
|
|
(2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
|
|
alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
|
|
time-to-first-token), each keyed only to the namespace its runtime exports.
|
|
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
|
|
relay, and `journalctl --user -u signal-relay` records successful
|
|
`sent to +12062476336:` lines for every rule in the set.
|
|
* **Dashboards: working.** Provisioned datasources — Prometheus (uid
|
|
`prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
|
|
`containers`, `observability` and `logs-overview` dashboards, all
|
|
installed from `shared/grafana/provisioning/`. The Loki datasource is how
|
|
you read logs: Loki has no web UI of its own, so Grafana Explore is the
|
|
human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
|
|
* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
|
|
after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
|
|
per-container CPU/memory here, and on this rootless-Podman host it works on one
|
|
condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
|
|
namespace, so without that flag cadvisor sees only its own cgroup root and
|
|
exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
|
|
names nothing itself — the libpod socket resolved only 2 of 34 rootless
|
|
containers — but the quadlet unit name is carried in the cgroup path
|
|
(`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
|
|
`container` label. Network and filesystem are NOT exported for these containers
|
|
(no netns/classid attribution under the host-PID layout), so the dashboard
|
|
graphs only CPU, memory and OOM and does not fake the rest.
|
|
* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
|
|
`deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
|
|
host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
|
|
own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
|
|
without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
|
|
`mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
|
|
emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
|
|
°C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
|
|
job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
|
|
`power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
|
|
the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
|
|
node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
|
|
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
|
|
a rootless Alloy cannot read the `0640` journal files, and the container-log
|
|
globs point at a root-storage layout that does not exist here. The fix is built
|
|
and tested in-repo (2026-10-20): Alloy is now a **root system service on
|
|
`Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
|
|
Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
|
|
lines and pushed 26,966 with zero drops. The **live** instance is still the old
|
|
rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
|
|
[[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
|
|
* **Not configured:** no blackbox exporter (the structural downs have no real
|
|
uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
|
|
not yet deployed (see *Logs* above).
|
|
|
|
## Citations
|
|
|
|
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
|
|
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
|
|
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
|
|
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
|
|
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
|
|
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
|
|
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
|
|
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
|