--- type: reference title: "Observability on Dankclaw" description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured." tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw] timestamp: 2026-09-11 --- # Observability on Dankclaw ## Summary Six containers and one host-side process: Prometheus (metrics), Grafana (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host metrics) and Alertmanager (alert delivery), with `signal-relay` on the host carrying every firing group to Signal. All of it is Quadlet-managed, boot-persistent and loopback-only, deployed by `scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works end to end; the log half is configured, healthy, and carries no data. This body was rewritten on 2026-10-18 from live evidence: the previous version was deleted down to its front matter by commit `93d1181` (the "OKF v0.1 compliance pass"), while sibling pages kept pointing at a "state note" on this page that no longer existed. Every statement below carries the command or file that proves it today. ## The stack | Service | Container | Published | Unit | Page | |---|---|---|---|---| | Prometheus | `docker.io/prom/prometheus:latest` | `127.0.0.1:9090` | `prometheus.container` | [[Prometheus]] | | Alertmanager | `docker.io/prom/alertmanager:latest` | `127.0.0.1:9093` | `alertmanager.container` | [[Alertmanager]] | | Grafana | `docker.io/grafana/grafana:latest` | `127.0.0.1:3002` → 3000 | `grafana.container` | [[Grafana]] | | Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] | | Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] | | node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] | | Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] | Every unit lives in `dankclaw/observability/quadlet/`, carries `Restart=always`/`RestartSec=10` and `WantedBy=default.target`, is installed at `~/.config/containers/systemd/.container`, and reports `generated` + `active` (`systemctl --user is-enabled|is-active` for all six, 2026-10-18; linger is on for `dank`). `signal-relay.service` reports `enabled` + `active` the same day, and `homepage-status.timer` — the root unit that probes all of these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies the probe registry `shared/homepage/services.json` to `/srv/homepage`). Dankserver runs one member of this family, node-exporter, as its own unit (`dankserver/observability/quadlet/node-exporter.container`, `generated` + `active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`). ## Exposure The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18) shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100` `127.0.0.1:12345` and no wildcard — because each of them is unauthenticated inside, and several carry lifecycle or query APIs. Browser access goes through Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard `anubis:8923`, and the post-auth handlers route by container name on `dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`) `grafana.franksdank.cloud` → `grafana:3000` (`:374-376`) `prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`). Alertmanager has no vhost at all. node-exporter is the exception that must live in the host network namespace, and it binds exactly two addresses rather than the wildcard — see [[Node-Exporter]]. The relay's bind is the one address a bridge container can reach on the host: `192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself and the podman bridge subnet `10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). ## Deployment `./scripts/deploy.sh dankclaw --service ` runs `deploy_observability` which installs the unit, copies that service's config (`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config` `copy_alloy_config`, `copy_grafana_provisioning`, plus `install_signal_relay` with the alertmanager deploy), then waits on the loopback health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy` `/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy` `/node-exporter`: `/metrics`). Startup order follows the data dependencies — `grafana` after `prometheus`, `alloy` after `loki` (`grafana.container:32-33`, `alloy.container:38-39`). See [[Observability Provisioning]] for what is provisioned as code and how to verify it through the APIs. ## Current state (2026-10-18) * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports eight up targets; the four downs are structural (the `comfyui` and `services` HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set: 4 alert rules + 2 recording rules (`/api/v1/rules`, matching `shared/prometheus/rules/llm-fleet.yml`). * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the relay, and `journalctl --user -u signal-relay` records successful `sent to +12062476336:` lines for every rule in the set. * **Dashboards: working.** One provisioned Prometheus datasource (uid `prometheus`) and the `llm-fleet` dashboard, both installed from `shared/grafana/provisioning/`. * **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both file globs refused) and Loki therefore holds zero streams — `/opt/loki/data` is a few hundred KB of WAL/compactor bookkeeping with **no chunk files** (8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns no label names. See [[Alloy]] → Delivery state and [[Loki]] → Contents. * **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the structural downs have no real uptime signal), no Alertmanager vhost. ## Citations - `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container` - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed) - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service` - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml` - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy` - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract) - Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`