Frank
e7f0d93c86
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
117 lines
7.0 KiB
Markdown
117 lines
7.0 KiB
Markdown
---
|
|
type: reference
|
|
title: "Observability on Dankclaw"
|
|
description: "The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured."
|
|
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
|
|
timestamp: 2026-09-11
|
|
---
|
|
|
|
# Observability on Dankclaw
|
|
|
|
## Summary
|
|
|
|
Six containers and one host-side process: Prometheus (metrics), Grafana
|
|
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
|
|
metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
|
|
carrying every firing group to Signal. All of it is Quadlet-managed,
|
|
boot-persistent and loopback-only, deployed by
|
|
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
|
|
end to end; the log half is configured, healthy, and carries no data.
|
|
|
|
This body was rewritten on 2026-10-18 from live evidence: the previous version
|
|
was deleted down to its front matter by commit `93d1181` (the "OKF v0.1
|
|
compliance pass"), while sibling pages kept pointing at a "state note" on this
|
|
page that no longer existed. Every statement below carries the command or file
|
|
that proves it today.
|
|
|
|
## The stack
|
|
|
|
| Service | Container | Published | Unit | Page |
|
|
|---|---|---|---|---|
|
|
| Prometheus | `docker.io/prom/prometheus:latest` | `127.0.0.1:9090` | `prometheus.container` | [[Prometheus]] |
|
|
| Alertmanager | `docker.io/prom/alertmanager:latest` | `127.0.0.1:9093` | `alertmanager.container` | [[Alertmanager]] |
|
|
| Grafana | `docker.io/grafana/grafana:latest` | `127.0.0.1:3002` → 3000 | `grafana.container` | [[Grafana]] |
|
|
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
|
|
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
|
|
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
|
|
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
|
|
|
|
Every unit lives in `dankclaw/observability/quadlet/`, carries
|
|
`Restart=always`/`RestartSec=10` and `WantedBy=default.target`, is installed at
|
|
`~/.config/containers/systemd/<name>.container`, and reports `generated` +
|
|
`active` (`systemctl --user is-enabled|is-active` for all six, 2026-10-18;
|
|
linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
|
|
the same day, and `homepage-status.timer` — the root unit that probes all of
|
|
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
|
|
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
|
|
Dankserver runs one member of this family, node-exporter, as its own unit
|
|
(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
|
|
`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
|
|
|
|
## Exposure
|
|
|
|
The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
|
|
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`,
|
|
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
|
|
inside, and several carry lifecycle or query APIs. Browser access goes through
|
|
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
|
|
`anubis:8923`, and the post-auth handlers route by container name on
|
|
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`),
|
|
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`),
|
|
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
|
|
Alertmanager has no vhost at all. node-exporter is the exception that must live
|
|
in the host network namespace, and it binds exactly two addresses rather than
|
|
the wildcard — see [[Node-Exporter]].
|
|
|
|
The relay's bind is the one address a bridge container can reach on the host:
|
|
`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
|
|
and the podman bridge subnet `10.89.0.0/16` before the DROP
|
|
(`scripts/deploy-remote.sh:255-262`).
|
|
|
|
## Deployment
|
|
|
|
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`,
|
|
which installs the unit, copies that service's config
|
|
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`,
|
|
`copy_alloy_config`, `copy_grafana_provisioning`, plus
|
|
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
|
|
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`,
|
|
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`,
|
|
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
|
|
`grafana` after `prometheus`, `alloy` after `loki`
|
|
(`grafana.container:32-33`, `alloy.container:38-39`). See
|
|
[[Observability Provisioning]] for what is provisioned as code and how to
|
|
verify it through the APIs.
|
|
|
|
## Current state (2026-10-18)
|
|
|
|
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
|
|
eight up targets; the four downs are structural (the `comfyui` and `services`
|
|
HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set:
|
|
4 alert rules + 2 recording rules (`/api/v1/rules`, matching
|
|
`shared/prometheus/rules/llm-fleet.yml`).
|
|
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
|
|
relay, and `journalctl --user -u signal-relay` records successful
|
|
`sent to +12062476336:` lines for every rule in the set.
|
|
* **Dashboards: working.** One provisioned Prometheus datasource (uid
|
|
`prometheus`) and the `llm-fleet` dashboard, both installed from
|
|
`shared/grafana/provisioning/`.
|
|
* **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both
|
|
file globs refused) and Loki therefore holds zero streams — `/opt/loki/data`
|
|
is a few hundred KB of WAL/compactor bookkeeping with **no chunk files**
|
|
(8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns
|
|
no label names.
|
|
See [[Alloy]] → Delivery state and [[Loki]] → Contents.
|
|
* **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the
|
|
structural downs have no real uptime signal), no Alertmanager vhost.
|
|
|
|
## Citations
|
|
|
|
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
|
|
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
|
|
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
|
|
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
|
|
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
|
|
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
|
|
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
|