Frank
ce370a5b28
iptables -I prepends, so the install block's final -I of the blanket DROP landed it at position 1, above all three ACCEPTs: every packet to 18093/tcp died there (live 2026-10-21: 159 packets on the DROP, 0 on each ACCEPT). The host's own health probe timed out beside a LISTENing socket and the whole dankclaw drain aborted at the alertmanager gate twice. Install now rebuilds the tail: delete any blanket DROP wherever it sits, keep the ACCEPTs at the head, append the DROP last - converges on hosts poisoned by the old order. New bats case runs install_signal_relay for real against an iptables sandbox seeded with the poisoned order (red on the old code, green on this one).
177 lines
12 KiB
Markdown
177 lines
12 KiB
Markdown
---
|
|
type: reference
|
|
title: "Observability on Dankclaw"
|
|
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
|
|
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
|
|
timestamp: 2026-10-20
|
|
---
|
|
|
|
# Observability on Dankclaw
|
|
|
|
## Summary
|
|
|
|
Seven containers and two host-side processes: Prometheus (metrics), Grafana
|
|
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
|
|
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
|
|
plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
|
|
boot-persistent and loopback-only, deployed by
|
|
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
|
|
end to end; the log half is configured, healthy, and carries no data.
|
|
|
|
This body was rewritten on 2026-10-18 from live evidence: the previous version
|
|
was deleted down to its front matter by commit `93d1181` (the "OKF v0.1
|
|
compliance pass"), while sibling pages kept pointing at a "state note" on this
|
|
page that no longer existed. Every statement below carries the command or file
|
|
that proves it today.
|
|
|
|
## The stack
|
|
|
|
| Service | Container | Published | Unit | Page |
|
|
|---|---|---|---|---|
|
|
| Prometheus | `docker.io/prom/prometheus:latest` | `127.0.0.1:9090` | `prometheus.container` | [[Prometheus]] |
|
|
| Alertmanager | `docker.io/prom/alertmanager:latest` | `127.0.0.1:9093` | `alertmanager.container` | [[Alertmanager]] |
|
|
| Grafana | `docker.io/grafana/grafana:latest` | `127.0.0.1:3002` → 3000 | `grafana.container` | [[Grafana]] |
|
|
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
|
|
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
|
|
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
|
|
| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
|
|
| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
|
|
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
|
|
|
|
Every unit lives in `dankclaw/observability/quadlet/`, carries
|
|
`Restart=always`/`RestartSec=10` and `WantedBy=default.target`, is installed at
|
|
`~/.config/containers/systemd/<name>.container`, and reports `generated` +
|
|
`active` (`systemctl --user is-enabled|is-active` for all six, 2026-10-18;
|
|
linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
|
|
the same day, and `homepage-status.timer` — the root unit that probes all of
|
|
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
|
|
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
|
|
Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
|
|
(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
|
|
`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
|
|
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
|
|
`gpu-exporter` (a user unit installed the same way as `signal-relay`).
|
|
|
|
## Exposure
|
|
|
|
The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
|
|
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`
|
|
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
|
|
inside, and several carry lifecycle or query APIs. Browser access goes through
|
|
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
|
|
`anubis:8923`, and the post-auth handlers route by container name on
|
|
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`)
|
|
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`)
|
|
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
|
|
Alertmanager has no vhost at all. node-exporter is the exception that must live
|
|
in the host network namespace, and it binds exactly two addresses rather than
|
|
the wildcard — see [[Node-Exporter]].
|
|
|
|
**Same-host reachability from a bridge container is measured, not inferred**
|
|
(2026-10-21, from inside the Prometheus container): `host.containers.internal`
|
|
(the podman gateway) reaches host services whether they bind host-net or are
|
|
published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
|
|
— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
|
|
every same-host Prometheus target uses the gateway and only cross-host targets
|
|
(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
|
|
relay at `192.168.0.143:18093` is reached from alertmanager's network only
|
|
while `install_signal_relay`'s INPUT rules are in force: loopback, the host
|
|
itself and the podman bridge subnet `10.89.0.0/16` accepted **above** a
|
|
trailing blanket DROP. The law the poisoned host taught (2026-10-21):
|
|
`iptables -I` prepends, so inserting the DROP last lands it first — the old
|
|
install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the
|
|
deployer's health gate failed beside a LISTENing socket, and the drain
|
|
aborted twice before `nft list ruleset` named the cause. The install now
|
|
rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head,
|
|
DROP appended); verify any change to it with the sandboxed bats case, not
|
|
with `-C` greps — `-C` answers existence, never order. Never point a scrape
|
|
target at a host LAN IP from its own host on the strength of one working
|
|
example; dial the gateway and probe.
|
|
|
|
## Deployment
|
|
|
|
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`
|
|
which installs the unit, copies that service's config
|
|
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`
|
|
`copy_alloy_config`, `copy_grafana_provisioning`, plus
|
|
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
|
|
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
|
|
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`
|
|
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
|
|
`grafana` after `prometheus`, `alloy` after `loki`
|
|
(`grafana.container:32-33`, `alloy.container:38-39`). See
|
|
[[Observability Provisioning]] for what is provisioned as code and how to
|
|
verify it through the APIs.
|
|
|
|
## Current state (2026-10-21)
|
|
|
|
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
|
|
every job up — the three LLM endpoints (frank-archon on `vllm:*`,
|
|
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
|
|
both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
|
|
the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
|
|
downs are retired — see [[Prometheus]] → Targets.
|
|
Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
|
|
(2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
|
|
alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
|
|
time-to-first-token), each keyed only to the namespace its runtime exports.
|
|
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
|
|
relay, and `journalctl --user -u signal-relay` records successful
|
|
`sent to +12062476336:` lines for every rule in the set.
|
|
* **Dashboards: working (after the 2026-10-21 copier fix).** Provisioned
|
|
datasources — Prometheus (uid `prometheus`) and Loki (uid `loki`) — and the
|
|
`llm-fleet`, `host-system`, `containers`, `observability` and
|
|
`logs-overview` boards. Until 2026-10-21 the copier carried only
|
|
`llm-fleet.json` by name and the `containers` board had never been
|
|
provisioned; the deployer now globs the directory. The Loki datasource is
|
|
how you read logs: Loki has no web UI of its own, so Grafana Explore is the
|
|
human surface (the guarded `loki.franksdank.cloud` vhost serves only the
|
|
API). Verify method and the copier law: [[Observability Provisioning]].
|
|
* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
|
|
after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
|
|
per-container CPU/memory here, and on this rootless-Podman host it works on one
|
|
condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
|
|
namespace, so without that flag cadvisor sees only its own cgroup root and
|
|
exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
|
|
names nothing itself — the libpod socket resolved only 2 of 34 rootless
|
|
containers — but the quadlet unit name is carried in the cgroup path
|
|
(`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
|
|
`container` label. Network and filesystem are NOT exported for these containers
|
|
(no netns/classid attribution under the host-PID layout), so the dashboard
|
|
graphs only CPU, memory and OOM and does not fake the rest.
|
|
* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
|
|
`deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
|
|
host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
|
|
own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
|
|
without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
|
|
`mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
|
|
emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
|
|
°C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
|
|
job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
|
|
`power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
|
|
the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
|
|
node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
|
|
* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
|
|
pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
|
|
the container-log globs pointed at a root-storage layout that does not exist
|
|
here) — fixed 2026-10-20 by making Alloy a **root system service on
|
|
`Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
|
|
but the stream carried only `job`/`service_name`, so every per-container
|
|
view was structurally empty: Loki metrics queries group only by INDEXED
|
|
stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
|
|
label (plus a `host` label) at ingestion — `loki.process` in
|
|
`shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
|
|
purpose-test law in [[Observability Provisioning]].
|
|
|
|
## Citations
|
|
|
|
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
|
|
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
|
|
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
|
|
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
|
|
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
|
|
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
|
|
- `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
|
|
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
|
|
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
|