serverAppConfigs/serverAppConfigs-docs/Infrastructure/Observability.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Observability on Dankclaw	The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured.	observability prometheus grafana loki alloy node-exporter quadlet dankclaw	2026-10-20

Observability on Dankclaw

Summary

Six containers and one host-side process: Prometheus (metrics), Grafana (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host metrics) and Alertmanager (alert delivery), with signal-relay on the host carrying every firing group to Signal. All of it is Quadlet-managed, boot-persistent and loopback-only, deployed by scripts/deploy-remote.sh → deploy_observability. The metrics half works end to end; the log half is configured, healthy, and carries no data.

This body was rewritten on 2026-10-18 from live evidence: the previous version was deleted down to its front matter by commit 93d1181 (the "OKF v0.1 compliance pass"), while sibling pages kept pointing at a "state note" on this page that no longer existed. Every statement below carries the command or file that proves it today.

The stack

Service	Container	Published	Unit	Page
Prometheus	docker.io/prom/prometheus:latest	127.0.0.1:9090	prometheus.container	Prometheus
Alertmanager	docker.io/prom/alertmanager:latest	127.0.0.1:9093	alertmanager.container	Alertmanager
Grafana	docker.io/grafana/grafana:latest	127.0.0.1:3002 → 3000	grafana.container	Grafana
Loki	localhost/franksdank/loki:latest	127.0.0.1:3100	loki.container	Loki
Alloy	localhost/franksdank/alloy:latest	127.0.0.1:12345	alloy.container	Alloy
node-exporter	docker.io/prom/node-exporter:latest	host netns: 127.0.0.1:9100 + 192.168.0.143:9100	node-exporter.container	Node-Exporter
Signal relay	host python3, not a container	192.168.0.143:18093	scripts/signal-relay.service (user unit)	Alertmanager

Every unit lives in dankclaw/observability/quadlet/, carries Restart=always/RestartSec=10 and WantedBy=default.target, is installed at ~/.config/containers/systemd/<name>.container, and reports generated + active (systemctl --user is-enabled|is-active for all six, 2026-10-18; linger is on for dank). signal-relay.service reports enabled + active the same day, and homepage-status.timer — the root unit that probes all of these on loopback — is enabled + active (deploy-remote.sh:319-321 copies the probe registry shared/homepage/services.json to /srv/homepage). Dankserver runs one member of this family, node-exporter, as its own unit (dankserver/observability/quadlet/node-exporter.container, generated + active, binding 127.0.0.1:9100 + 192.168.0.235:9100).

Exposure

The five HTTP surfaces publish on 127.0.0.1 only — ss -ltnp (2026-10-18) shows 127.0.0.1:9090, 127.0.0.1:9093, 127.0.0.1:3002, 127.0.0.1:3100 127.0.0.1:12345 and no wildcard — because each of them is unauthenticated inside, and several carry lifecycle or query APIs. Browser access goes through Caddy: each vhost (dankclaw/Caddyfile:128-145) proxies to the guard anubis:8923, and the post-auth handlers route by container name on dankclaw_default — loki.franksdank.cloud → loki:3100 (:364-366) grafana.franksdank.cloud → grafana:3000 (:374-376) prometheus.franksdank.cloud → prometheus:9090 (:379-381). Alertmanager has no vhost at all. node-exporter is the exception that must live in the host network namespace, and it binds exactly two addresses rather than the wildcard — see Node-Exporter.

The relay's bind is the one address a bridge container can reach on the host: 192.168.0.143:18093, with iptables INPUT admitting loopback, the host itself and the podman bridge subnet 10.89.0.0/16 before the DROP (scripts/deploy-remote.sh:255-262).

Deployment

./scripts/deploy.sh dankclaw --service <name> runs deploy_observability which installs the unit, copies that service's config (copy_prometheus_config, copy_alertmanager_config, copy_loki_config copy_alloy_config, copy_grafana_provisioning, plus install_signal_relay with the alertmanager deploy), then waits on the loopback health endpoint (/prometheus: /-/healthy, /alertmanager: /-/healthy /grafana: /api/health, /loki: /ready, /alloy: /-/healthy /node-exporter: /metrics). Startup order follows the data dependencies — grafana after prometheus, alloy after loki (grafana.container:32-33, alloy.container:38-39). See Observability Provisioning for what is provisioned as code and how to verify it through the APIs.

Current state (2026-10-20)

• Metrics: working. curl -s http://127.0.0.1:9090/api/v1/targets reports eight up targets — the three LLM endpoints (frank-archon on vllm:*, frank-dragoon and frank-zealot on llamacpp:*), grafana, loki, prometheus and both node exporters. The four downs are structural (the comfyui and services HTML probes, and Gitea's disabled metrics endpoint), not outages. Two rule groups in shared/prometheus/rules/llm-fleet.yml: llm-fleet (2 recording + 4 alert, llamacpp:*) and vllm-fleet (1 recording + 4 alert, vllm:* — archon queueing, prefix collapse, preemption, p95 time-to-first-token), each keyed only to the namespace its runtime exports.
• Delivery: working. Alertmanager's single signal receiver webhooks the relay, and journalctl --user -u signal-relay records successful sent to +12062476336: lines for every rule in the set.
• Dashboards: working. Provisioned datasources — Prometheus (uid prometheus) and Loki (uid loki) — and the llm-fleet dashboard, all installed from shared/grafana/provisioning/. The Loki datasource is how you read logs: Loki has no web UI of its own, so Grafana Explore is the human surface (the guarded loki.franksdank.cloud vhost serves only the API).
• Logs: fixed in code, awaiting deploy. The pipeline shipped nothing because a rootless Alloy cannot read the 0640 journal files, and the container-log globs point at a root-storage layout that does not exist here. The fix is built and tested in-repo (2026-10-20): Alloy is now a root system service on Network=host, reading system.journal + user-*.journal and pushing to Loki's 127.0.0.1:3100 loopback publish — a throwaway root probe read 31,825 lines and pushed 26,966 with zero drops. The live instance is still the old rootless unit until deploy.sh dankclaw --service alloy runs. Detail in Alloy → Why root; Loki's store state in Loki → Contents.
• Not configured: no blackbox exporter (the structural downs have no real uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but not yet deployed (see Logs above).

Citations

• dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container
• scripts/deploy-remote.sh:219-300 (copiers + relay), :521-536 (health waits), :477-486 (grafana env seed)
• scripts/signal-relay.py:26-49, scripts/signal-relay.service
• shared/prometheus/prometheus.yml, shared/prometheus/alertmanager.yml, shared/prometheus/rules/llm-fleet.yml
• shared/grafana/provisioning/, shared/loki/loki-config.yaml, shared/alloy/config.alloy
• tests/observability_quadlet.bats (boot persistence, no wildcard publish, probe addresses, delivery contract)
• Host evidence 2026-10-18: ss -ltnp, podman ps, systemctl --user is-enabled|is-active, curl on 127.0.0.1:9090/api/v1/{targets,rules,alerts}, 127.0.0.1:3100/loki/api/v1/labels, 127.0.0.1:12345/metrics, 127.0.0.1:3002/api/health, 192.168.0.143:18093/health, journalctl --user -u signal-relay