Frank
e7f0d93c86
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
7.0 KiB
type, title, description, tags, timestamp
| type | title | description | tags | timestamp | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| reference | Observability on Dankclaw | The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured. |
|
2026-09-11 |
Observability on Dankclaw
Summary
Six containers and one host-side process: Prometheus (metrics), Grafana
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
metrics) and Alertmanager (alert delivery), with signal-relay on the host
carrying every firing group to Signal. All of it is Quadlet-managed,
boot-persistent and loopback-only, deployed by
scripts/deploy-remote.sh → deploy_observability. The metrics half works
end to end; the log half is configured, healthy, and carries no data.
This body was rewritten on 2026-10-18 from live evidence: the previous version
was deleted down to its front matter by commit 93d1181 (the "OKF v0.1
compliance pass"), while sibling pages kept pointing at a "state note" on this
page that no longer existed. Every statement below carries the command or file
that proves it today.
The stack
| Service | Container | Published | Unit | Page |
|---|---|---|---|---|
| Prometheus | docker.io/prom/prometheus:latest |
127.0.0.1:9090 |
prometheus.container |
Prometheus |
| Alertmanager | docker.io/prom/alertmanager:latest |
127.0.0.1:9093 |
alertmanager.container |
Alertmanager |
| Grafana | docker.io/grafana/grafana:latest |
127.0.0.1:3002 → 3000 |
grafana.container |
Grafana |
| Loki | localhost/franksdank/loki:latest |
127.0.0.1:3100 |
loki.container |
Loki |
| Alloy | localhost/franksdank/alloy:latest |
127.0.0.1:12345 |
alloy.container |
Alloy |
| node-exporter | docker.io/prom/node-exporter:latest |
host netns: 127.0.0.1:9100 + 192.168.0.143:9100 |
node-exporter.container |
Node-Exporter |
| Signal relay | host python3, not a container |
192.168.0.143:18093 |
scripts/signal-relay.service (user unit) |
Alertmanager |
Every unit lives in dankclaw/observability/quadlet/, carries
Restart=always/RestartSec=10 and WantedBy=default.target, is installed at
~/.config/containers/systemd/<name>.container, and reports generated +
active (systemctl --user is-enabled|is-active for all six, 2026-10-18;
linger is on for dank). signal-relay.service reports enabled + active
the same day, and homepage-status.timer — the root unit that probes all of
these on loopback — is enabled + active (deploy-remote.sh:319-321 copies
the probe registry shared/homepage/services.json to /srv/homepage).
Dankserver runs one member of this family, node-exporter, as its own unit
(dankserver/observability/quadlet/node-exporter.container, generated +
active, binding 127.0.0.1:9100 + 192.168.0.235:9100).
Exposure
The five HTTP surfaces publish on 127.0.0.1 only — ss -ltnp (2026-10-18)
shows 127.0.0.1:9090, 127.0.0.1:9093, 127.0.0.1:3002, 127.0.0.1:3100,
127.0.0.1:12345 and no wildcard — because each of them is unauthenticated
inside, and several carry lifecycle or query APIs. Browser access goes through
Caddy: each vhost (dankclaw/Caddyfile:128-145) proxies to the guard
anubis:8923, and the post-auth handlers route by container name on
dankclaw_default — loki.franksdank.cloud → loki:3100 (:364-366),
grafana.franksdank.cloud → grafana:3000 (:374-376),
prometheus.franksdank.cloud → prometheus:9090 (:379-381).
Alertmanager has no vhost at all. node-exporter is the exception that must live
in the host network namespace, and it binds exactly two addresses rather than
the wildcard — see Node-Exporter.
The relay's bind is the one address a bridge container can reach on the host:
192.168.0.143:18093, with iptables INPUT admitting loopback, the host itself
and the podman bridge subnet 10.89.0.0/16 before the DROP
(scripts/deploy-remote.sh:255-262).
Deployment
./scripts/deploy.sh dankclaw --service <name> runs deploy_observability,
which installs the unit, copies that service's config
(copy_prometheus_config, copy_alertmanager_config, copy_loki_config,
copy_alloy_config, copy_grafana_provisioning, plus
install_signal_relay with the alertmanager deploy), then waits on the loopback
health endpoint (/prometheus: /-/healthy, /alertmanager: /-/healthy,
/grafana: /api/health, /loki: /ready, /alloy: /-/healthy,
/node-exporter: /metrics). Startup order follows the data dependencies —
grafana after prometheus, alloy after loki
(grafana.container:32-33, alloy.container:38-39). See
Observability Provisioning for what is provisioned as code and how to
verify it through the APIs.
Current state (2026-10-18)
- Metrics: working.
curl -s http://127.0.0.1:9090/api/v1/targetsreports eight up targets; the four downs are structural (thecomfyuiandservicesHTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set: 4 alert rules + 2 recording rules (/api/v1/rules, matchingshared/prometheus/rules/llm-fleet.yml). - Delivery: working. Alertmanager's single
signalreceiver webhooks the relay, andjournalctl --user -u signal-relayrecords successfulsent to +12062476336:lines for every rule in the set. - Dashboards: working. One provisioned Prometheus datasource (uid
prometheus) and thellm-fleetdashboard, both installed fromshared/grafana/provisioning/. - Logs: configured, empty. Alloy ships zero lines (journal counter 0, both
file globs refused) and Loki therefore holds zero streams —
/opt/loki/datais a few hundred KB of WAL/compactor bookkeeping with no chunk files (8 files total byfind, 2026-10-18) andGET /loki/api/v1/labelsreturns no label names. See Alloy → Delivery state and Loki → Contents. - Not configured: no Loki datasource in Grafana, no blackbox exporter (the structural downs have no real uptime signal), no Alertmanager vhost.
Citations
dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.containerscripts/deploy-remote.sh:219-300(copiers + relay),:521-536(health waits),:477-486(grafana env seed)scripts/signal-relay.py:26-49,scripts/signal-relay.serviceshared/prometheus/prometheus.yml,shared/prometheus/alertmanager.yml,shared/prometheus/rules/llm-fleet.ymlshared/grafana/provisioning/,shared/loki/loki-config.yaml,shared/alloy/config.alloytests/observability_quadlet.bats(boot persistence, no wildcard publish, probe addresses, delivery contract)- Host evidence 2026-10-18:
ss -ltnp,podman ps,systemctl --user is-enabled|is-active,curlon127.0.0.1:9090/api/v1/{targets,rules,alerts},127.0.0.1:3100/loki/api/v1/labels,127.0.0.1:12345/metrics,127.0.0.1:3002/api/health,192.168.0.143:18093/health,journalctl --user -u signal-relay