serverAppConfigs/serverAppConfigs-docs/Infrastructure/Observability.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Observability on Dankclaw	The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured.	observability prometheus grafana loki alloy node-exporter gpu-exporter quadlet dankclaw	2026-10-20

Observability on Dankclaw

Summary

Seven containers and two host-side processes: Prometheus (metrics), Grafana (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), plus two host python3 processes — gpu-exporter (per-GPU metrics) and signal-relay (Signal delivery). The containers are all Quadlet-managed, boot-persistent and loopback-only, deployed by scripts/deploy-remote.sh → deploy_observability. The metrics half works end to end; the log half is configured, healthy, and carries no data.

This body was rewritten on 2026-10-18 from live evidence: the previous version was deleted down to its front matter by commit 93d1181 (the "OKF v0.1 compliance pass"), while sibling pages kept pointing at a "state note" on this page that no longer existed. Every statement below carries the command or file that proves it today.

The stack

Service	Container	Published	Unit	Page
Prometheus	docker.io/prom/prometheus:latest	127.0.0.1:9090	prometheus.container	Prometheus
Alertmanager	docker.io/prom/alertmanager:latest	127.0.0.1:9093	alertmanager.container	Alertmanager
Grafana	docker.io/grafana/grafana:latest	127.0.0.1:3002 → 3000	grafana.container	Grafana
Loki	localhost/franksdank/loki:latest	127.0.0.1:3100	loki.container	Loki
Alloy	localhost/franksdank/alloy:latest	127.0.0.1:12345	alloy.container	Alloy
node-exporter	docker.io/prom/node-exporter:latest	host netns: 127.0.0.1:9100 + 192.168.0.143:9100	node-exporter.container	Node-Exporter
cadvisor	gcr.io/cadvisor/cadvisor:v0.49.1	bridge publish 127.0.0.1:8080 + 192.168.0.143:8080	cadvisor.container	—
GPU exporter	host python3, not a container	127.0.0.1:9401 + host LAN :9401	scripts/gpu-exporter.service (user unit)	—
Signal relay	host python3, not a container	192.168.0.143:18093	scripts/signal-relay.service (user unit)	Alertmanager

Every unit lives in dankclaw/observability/quadlet/, carries Restart=always/RestartSec=10 and WantedBy=default.target, is installed at ~/.config/containers/systemd/<name>.container, and reports generated + active (systemctl --user is-enabled|is-active for all six, 2026-10-18; linger is on for dank). signal-relay.service reports enabled + active the same day, and homepage-status.timer — the root unit that probes all of these on loopback — is enabled + active (deploy-remote.sh:319-321 copies the probe registry shared/homepage/services.json to /srv/homepage). Dankserver runs the same family: node-exporter and cadvisor as its own quadlets (dankserver/observability/quadlet/{node-exporter,cadvisor}.container, generated + active; node-exporter binds 127.0.0.1:9100 + 192.168.0.235:9100, cadvisor publishes 127.0.0.1:8080 + 192.168.0.235:8080), plus the host-process gpu-exporter (a user unit installed the same way as signal-relay).

Exposure

The five HTTP surfaces publish on 127.0.0.1 only — ss -ltnp (2026-10-18) shows 127.0.0.1:9090, 127.0.0.1:9093, 127.0.0.1:3002, 127.0.0.1:3100 127.0.0.1:12345 and no wildcard — because each of them is unauthenticated inside, and several carry lifecycle or query APIs. Browser access goes through Caddy: each vhost (dankclaw/Caddyfile:128-145) proxies to the guard anubis:8923, and the post-auth handlers route by container name on dankclaw_default — loki.franksdank.cloud → loki:3100 (:364-366) grafana.franksdank.cloud → grafana:3000 (:374-376) prometheus.franksdank.cloud → prometheus:9090 (:379-381). Alertmanager has no vhost at all. node-exporter is the exception that must live in the host network namespace, and it binds exactly two addresses rather than the wildcard — see Node-Exporter.

The relay's bind is the one address a bridge container can reach on the host: 192.168.0.143:18093, with iptables INPUT admitting loopback, the host itself and the podman bridge subnet 10.89.0.0/16 before the DROP (scripts/deploy-remote.sh:255-262).

Deployment

./scripts/deploy.sh dankclaw --service <name> runs deploy_observability which installs the unit, copies that service's config (copy_prometheus_config, copy_alertmanager_config, copy_loki_config copy_alloy_config, copy_grafana_provisioning, plus install_signal_relay with the alertmanager deploy), then waits on the loopback health endpoint (/prometheus: /-/healthy, /alertmanager: /-/healthy /grafana: /api/health, /loki: /ready, /alloy: /-/healthy /node-exporter: /metrics). Startup order follows the data dependencies — grafana after prometheus, alloy after loki (grafana.container:32-33, alloy.container:38-39). See Observability Provisioning for what is provisioned as code and how to verify it through the APIs.

Current state (2026-10-20)

• Metrics: working. curl -s http://127.0.0.1:9090/api/v1/targets reports eight up targets — the three LLM endpoints (frank-archon on vllm:*, frank-dragoon and frank-zealot on llamacpp:*), grafana, loki, prometheus and both node exporters. The four downs are structural (the comfyui and services HTML probes, and Gitea's disabled metrics endpoint), not outages. Two rule groups in shared/prometheus/rules/llm-fleet.yml: llm-fleet (2 recording + 4 alert, llamacpp:*) and vllm-fleet (1 recording + 4 alert, vllm:* — archon queueing, prefix collapse, preemption, p95 time-to-first-token), each keyed only to the namespace its runtime exports.
• Delivery: working. Alertmanager's single signal receiver webhooks the relay, and journalctl --user -u signal-relay records successful sent to +12062476336: lines for every rule in the set.
• Dashboards: working. Provisioned datasources — Prometheus (uid prometheus) and Loki (uid loki) — and the llm-fleet, host-system, containers, observability and logs-overview dashboards, all installed from shared/grafana/provisioning/. The Loki datasource is how you read logs: Loki has no web UI of its own, so Grafana Explore is the human surface (the guarded loki.franksdank.cloud vhost serves only the API).
• Per-container metrics: the containers dashboard (added 2026-10-20; live after deploy.sh <host> --service cadvisor). cadvisor is the only way to see per-container CPU/memory here, and on this rootless-Podman host it works on one condition: --cgroupns=host. Podman starts every container in a PRIVATE cgroup namespace, so without that flag cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor names nothing itself — the libpod socket resolved only 2 of 34 rootless containers — but the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/…), so prometheus.yml relabels that path into a container label. Network and filesystem are NOT exported for these containers (no netns/classid attribution under the host-PID layout), so the dashboard graphs only CPU, memory and OOM and does not fake the rest.
• Per-GPU metrics: the host-system GPU panels (added 2026-10-20; live after deploy.sh <host> --service gpu-exporter). scripts/gpu-exporter.py is a host-process user unit on BOTH hosts — not a quadlet, because it reads the host's own DRM sysfs and shells to nvidia-smi, neither of which a container can do without GPU passthrough. It reads each AMD card's gpu_busy_percent / mem_info_vram_* / hwmon temp1_input and one nvidia-smi --query-gpu CSV, and emits unified gpu_* families (utilization %, VRAM used/total bytes, temperature °C, power W) labelled gpu / vendor / model, with host added by the scrape job. Two laws: absent means absent, never a zero — the R9700s expose no sysfs power1_average, so they carry no gpu_power_watts line at all (only NVIDIA and the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
• Logs: fixed in code, awaiting deploy. The pipeline shipped nothing because a rootless Alloy cannot read the 0640 journal files, and the container-log globs point at a root-storage layout that does not exist here. The fix is built and tested in-repo (2026-10-20): Alloy is now a root system service on Network=host, reading system.journal + user-*.journal and pushing to Loki's 127.0.0.1:3100 loopback publish — a throwaway root probe read 31,825 lines and pushed 26,966 with zero drops. The live instance is still the old rootless unit until deploy.sh dankclaw --service alloy runs. Detail in Alloy → Why root; Loki's store state in Loki → Contents.
• Not configured: no blackbox exporter (the structural downs have no real uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but not yet deployed (see Logs above).

Citations

• dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container
• scripts/deploy-remote.sh:219-300 (copiers + relay), :521-536 (health waits), :477-486 (grafana env seed)
• scripts/signal-relay.py:26-49, scripts/signal-relay.service
• scripts/gpu-exporter.py, scripts/gpu-exporter.service (per-GPU host exporter)
• shared/prometheus/prometheus.yml, shared/prometheus/alertmanager.yml, shared/prometheus/rules/llm-fleet.yml
• shared/grafana/provisioning/, shared/loki/loki-config.yaml, shared/alloy/config.alloy
• tests/observability_quadlet.bats (boot persistence, no wildcard publish, probe addresses, delivery contract)
• Host evidence 2026-10-18: ss -ltnp, podman ps, systemctl --user is-enabled|is-active, curl on 127.0.0.1:9090/api/v1/{targets,rules,alerts}, 127.0.0.1:3100/loki/api/v1/labels, 127.0.0.1:12345/metrics, 127.0.0.1:3002/api/health, 192.168.0.143:18093/health, journalctl --user -u signal-relay