serverAppConfigs/serverAppConfigs-docs/Infrastructure/Observability.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Observability on Dankclaw	The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured.	observability prometheus grafana loki alloy node-exporter gpu-exporter quadlet dankclaw	2026-10-20

Observability on Dankclaw

Summary

Seven containers and two host-side processes: Prometheus (metrics), Grafana (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), plus two host python3 processes — gpu-exporter (per-GPU metrics) and signal-relay (Signal delivery). The containers are all Quadlet-managed, boot-persistent and loopback-only, deployed by scripts/deploy-remote.sh → deploy_observability. The metrics half works end to end; the log half is configured, healthy, and carries no data.

This body was rewritten on 2026-10-18 from live evidence: the previous version was deleted down to its front matter by commit 93d1181 (the "OKF v0.1 compliance pass"), while sibling pages kept pointing at a "state note" on this page that no longer existed. Every statement below carries the command or file that proves it today.

The stack

Service	Container	Published	Unit	Page
Prometheus	docker.io/prom/prometheus:latest	127.0.0.1:9090	prometheus.container	Prometheus
Alertmanager	docker.io/prom/alertmanager:latest	127.0.0.1:9093	alertmanager.container	Alertmanager
Grafana	docker.io/grafana/grafana:latest	127.0.0.1:3002 → 3000	grafana.container	Grafana
Loki	localhost/franksdank/loki:latest	127.0.0.1:3100	loki.container	Loki
Alloy	localhost/franksdank/alloy:latest	127.0.0.1:12345	alloy.container	Alloy
node-exporter	docker.io/prom/node-exporter:latest	host netns: 127.0.0.1:9100 + 192.168.0.143:9100	node-exporter.container	Node-Exporter
cadvisor	gcr.io/cadvisor/cadvisor:v0.49.1	bridge publish 127.0.0.1:8080 + 192.168.0.143:8080	cadvisor.container	—
GPU exporter	host python3, not a container	127.0.0.1:9401 + host LAN :9401	scripts/gpu-exporter.service (user unit)	—
Signal relay	host python3, not a container	192.168.0.143:18093	scripts/signal-relay.service (user unit)	Alertmanager

Every unit lives in dankclaw/observability/quadlet/, carries Restart=always/RestartSec=10 and WantedBy=default.target, is installed at ~/.config/containers/systemd/<name>.container, and reports generated + active (systemctl --user is-enabled|is-active for all six, 2026-10-18; linger is on for dank). signal-relay.service reports enabled + active the same day, and homepage-status.timer — the root unit that probes all of these on loopback — is enabled + active (deploy-remote.sh:319-321 copies the probe registry shared/homepage/services.json to /srv/homepage). Dankserver runs the same family: node-exporter and cadvisor as its own quadlets (dankserver/observability/quadlet/{node-exporter,cadvisor}.container, generated + active; node-exporter binds 127.0.0.1:9100 + 192.168.0.235:9100, cadvisor publishes 127.0.0.1:8080 + 192.168.0.235:8080), plus the host-process gpu-exporter (a user unit installed the same way as signal-relay).

Exposure

The five HTTP surfaces publish on 127.0.0.1 only — ss -ltnp (2026-10-18) shows 127.0.0.1:9090, 127.0.0.1:9093, 127.0.0.1:3002, 127.0.0.1:3100 127.0.0.1:12345 and no wildcard — because each of them is unauthenticated inside, and several carry lifecycle or query APIs. Browser access goes through Caddy: each vhost (dankclaw/Caddyfile:128-145) proxies to the guard anubis:8923, and the post-auth handlers route by container name on dankclaw_default — loki.franksdank.cloud → loki:3100 (:364-366) grafana.franksdank.cloud → grafana:3000 (:374-376) prometheus.franksdank.cloud → prometheus:9090 (:379-381). Alertmanager has no vhost at all. node-exporter is the exception that must live in the host network namespace, and it binds exactly two addresses rather than the wildcard — see Node-Exporter.

Same-host reachability from a bridge container is measured, not inferred (2026-10-21, from inside the Prometheus container): host.containers.internal (the podman gateway) reaches host services whether they bind host-net or are published ports — node :9100, cadvisor :8080, gpu :9401 all answer there — while the host's own LAN IP is refused (192.168.0.143:8080, :9401), so every same-host Prometheus target uses the gateway and only cross-host targets (dankserver) use plain LAN. Whether LAN works at all is network-specific: the relay at 192.168.0.143:18093 is reached from alertmanager's network only while install_signal_relay's INPUT rules are in force: loopback, the host itself and the podman bridge subnet 10.89.0.0/16 accepted above a trailing blanket DROP. The law the poisoned host taught (2026-10-21): iptables -I prepends, so inserting the DROP last lands it first — the old install order dropped every packet (159 on the DROP, 0 on the ACCEPTs), the deployer's health gate failed beside a LISTENing socket, and the drain aborted twice before nft list ruleset named the cause. The install now rebuilds the tail (delete the DROP wherever it sits, ACCEPTs at the head, DROP appended); verify any change to it with the sandboxed bats case, not with -C greps — -C answers existence, never order. Never point a scrape target at a host LAN IP from its own host on the strength of one working example; dial the gateway and probe.

Deployment

./scripts/deploy.sh dankclaw --service <name> runs deploy_observability which installs the unit, copies that service's config (copy_prometheus_config, copy_alertmanager_config, copy_loki_config copy_alloy_config, copy_grafana_provisioning, plus install_signal_relay with the alertmanager deploy), then waits on the loopback health endpoint (/prometheus: /-/healthy, /alertmanager: /-/healthy /grafana: /api/health, /loki: /ready, /alloy: /-/healthy /node-exporter: /metrics). Startup order follows the data dependencies — grafana after prometheus, alloy after loki (grafana.container:32-33, alloy.container:38-39). See Observability Provisioning for what is provisioned as code and how to verify it through the APIs.

Current state (2026-10-21)

• Metrics: working. curl -s http://127.0.0.1:9090/api/v1/targets reports every job up — the three LLM endpoints (frank-archon on vllm:*, frank-dragoon and frank-zealot on llamacpp:*), grafana, loki, prometheus, both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and the blackbox-probed services (SearXNG, OpenWebUI, ComfyUI). The structural downs are retired — see Prometheus → Targets. Two rule groups in shared/prometheus/rules/llm-fleet.yml: llm-fleet (2 recording + 4 alert, llamacpp:*) and vllm-fleet (1 recording + 4 alert, vllm:* — archon queueing, prefix collapse, preemption, p95 time-to-first-token), each keyed only to the namespace its runtime exports.
• Delivery: working. Alertmanager's single signal receiver webhooks the relay, and journalctl --user -u signal-relay records successful sent to +12062476336: lines for every rule in the set.
• Dashboards: working (after the 2026-10-21 copier fix). Provisioned datasources — Prometheus (uid prometheus) and Loki (uid loki) — and the llm-fleet, host-system, containers, observability and logs-overview boards. Until 2026-10-21 the copier carried only llm-fleet.json by name and the containers board had never been provisioned; the deployer now globs the directory. The Loki datasource is how you read logs: Loki has no web UI of its own, so Grafana Explore is the human surface (the guarded loki.franksdank.cloud vhost serves only the API). Verify method and the copier law: Observability Provisioning.
• Per-container metrics: the containers dashboard (added 2026-10-20; live after deploy.sh <host> --service cadvisor). cadvisor is the only way to see per-container CPU/memory here, and on this rootless-Podman host it works on one condition: --cgroupns=host. Podman starts every container in a PRIVATE cgroup namespace, so without that flag cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor names nothing itself — the libpod socket resolved only 2 of 34 rootless containers — but the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/…), so prometheus.yml relabels that path into a container label. Network and filesystem are NOT exported for these containers (no netns/classid attribution under the host-PID layout), so the dashboard graphs only CPU, memory and OOM and does not fake the rest.
• Per-GPU metrics: the host-system GPU panels (added 2026-10-20; live after deploy.sh <host> --service gpu-exporter). scripts/gpu-exporter.py is a host-process user unit on BOTH hosts — not a quadlet, because it reads the host's own DRM sysfs and shells to nvidia-smi, neither of which a container can do without GPU passthrough. It reads each AMD card's gpu_busy_percent / mem_info_vram_* / hwmon temp1_input and one nvidia-smi --query-gpu CSV, and emits unified gpu_* families (utilization %, VRAM used/total bytes, temperature °C, power W) labelled gpu / vendor / model, with host added by the scrape job. Two laws: absent means absent, never a zero — the R9700s expose no sysfs power1_average, so they carry no gpu_power_watts line at all (only NVIDIA and the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
• Logs: flowing; container labels land with the 2026-10-21 deploy. The pipeline once shipped nothing (rootless Alloy cannot read the 0640 journal; the container-log globs pointed at a root-storage layout that does not exist here) — fixed 2026-10-20 by making Alloy a root system service on Network=host. Journal lines now reach Loki (measured ~6.5k lines/5min), but the stream carried only job/service_name, so every per-container view was structurally empty: Loki metrics queries group only by INDEXED stream labels. Alloy now promotes podman's CONTAINER_NAME to a container label (plus a host label) at ingestion — loki.process in shared/alloy/config.alloy; detail in Alloy → Why root, and the purpose-test law in Observability Provisioning.

Citations

• dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container
• scripts/deploy-remote.sh:219-300 (copiers + relay), :521-536 (health waits), :477-486 (grafana env seed)
• scripts/signal-relay.py:26-49, scripts/signal-relay.service
• scripts/gpu-exporter.py, scripts/gpu-exporter.service (per-GPU host exporter)
• shared/prometheus/prometheus.yml, shared/prometheus/alertmanager.yml, shared/prometheus/rules/llm-fleet.yml
• shared/grafana/provisioning/, shared/loki/loki-config.yaml, shared/alloy/config.alloy
• shared/blackbox/blackbox.yml (prober modules; the listen bind is on the unit's Exec)
• tests/observability_quadlet.bats (boot persistence, no wildcard publish, probe addresses, delivery contract)
• Host evidence 2026-10-18: ss -ltnp, podman ps, systemctl --user is-enabled|is-active, curl on 127.0.0.1:9090/api/v1/{targets,rules,alerts}, 127.0.0.1:3100/loki/api/v1/labels, 127.0.0.1:12345/metrics, 127.0.0.1:3002/api/health, 192.168.0.143:18093/health, journalctl --user -u signal-relay