serverAppConfigs/serverAppConfigs-docs/Infrastructure/Observability.md

---

type: reference title: Observability on Dankclaw description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy and node-exporter: what runs, how it is deployed, how it is exposed, and what is still unconfigured. tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw] timestamp: 2026-09-11

Observability

The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards and node-exporter for host metrics. This is a small homelab — no Kubernetes.

Stack

graph LR
    S[Services] -->|/metrics| P[Prometheus]
    H[Host] -->|":9100"| P
    S -->|logs| A[Alloy]
    J[journald] --> A
    A --> L[Loki]
    P --> G[Grafana]
    L --> G

Component	Role	Image	Published
Prometheus	metrics collection	prom/prometheus	127.0.0.1:9090
Grafana	dashboards	grafana/grafana	127.0.0.1:3002
Loki	log storage & query	franksdank/loki (local build)	127.0.0.1:3100
Alloy	log shipping	franksdank/alloy (local build)	127.0.0.1:12345
Node-Exporter	host metrics	prom/node-exporter	192.168.0.143:9100, host netns

Current state: installed, healthy, and mostly unconfigured

Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:

• Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither. shared/grafana/provisioning/ is installed to /opt/grafana/provisioning by the deployer and read by the container: the Prometheus datasource (http://prometheus:9090, fixed uid prometheus) and the LLM fleet — capacity and reuse dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers vllm:kv_cache_usage_perc for frank-medium, and GET /api/search?type=dash-db returns uid llm-fleet. One trap worth knowing: a provisioned dashboard does not appear in the legacy dashboard table on this Grafana version (it lives in unified storage behind dashboard.grafana.app), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
• Prometheus has no alerting at all. /opt/prometheus/config/ contains prometheus.yml and nothing else — no rule files, no recording rules — and there is no Alertmanager anywhere in the stack. It scrapes and retains; it never notifies anyone.
• Loki is configured but unread. Storage is filesystem with 30-day retention, auth_enabled: false, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
• node-exporter runs its default collector set — no collector tuning, no textfile collector for custom metrics.
• Alloy is the one configured piece. It ships the host journal and podman container logs to Loki with job labels, and that pipeline is live.

The honest summary: metrics and logs are collected and retained, one dashboard now visualizes the LLM fleet, and nothing is alerted on — there is no rules file and no Alertmanager anywhere, so every graph in this stack is read after the fact by a human, and correlating metrics with logs is still manual.

Deployment

All five are rootless Quadlet units on Dankclaw. The units live in dankclaw/observability/quadlet/<name>.container and install to ~/.config/containers/systemd/, generating <name>.service.

They left dankclaw/compose.yaml on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots.

./scripts/deploy.sh dankclaw --service loki        # or prometheus|grafana|alloy|node-exporter

deploy_observability in scripts/deploy-remote.sh installs the unit, restarts it and waits on its endpoint. Ordering lives in the units themselves (Grafana wants Prometheus, Alloy wants Loki), and SERVICE_ORDER in the planner puts each dependency first.

Every unit carries a healthcheck, so a wedged process reads unhealthy in podman ps rather than merely Up. See Loki for the one unit that cannot declare it inline.

Exposure

The rule is the same as for every other service: the only off-host path is a guarded vhost (<name>.franksdank.cloud → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong:

• Prometheus, Grafana, Loki and Alloy publish on 127.0.0.1 only. Caddy reaches them by container name over the dankclaw_default network, and the host-side homepage status timer probes loopback. Publishing on 0.0.0.0 — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
• node-exporter is the one deliberate LAN listener. Prometheus runs in a container and cannot reach the host's loopback, so the unit binds exactly two addresses: 127.0.0.1:9100 for the homepage status probe and 192.168.0.143:9100 for Prometheus. It authenticates nothing, so it never gets a vhost.

Scrape surface

shared/prometheus/prometheus.yml is the source of truth (installed to /opt/prometheus/config/ on every deploy). It scrapes:

Job	Target	Notes
prometheus	localhost:9090	itself
node-dankclaw, node-dankserver	192.168.0.143:9100, 192.168.0.235:9100	one node-exporter per host
gitea	192.168.0.143:3000	published LAN port (Gitea is on an isolated network)
grafana, loki	grafana:3000, loki:3100	container names on dankclaw_default
comfyui	comfyui:8188	UI probe, no native exporter
services	192.168.0.143:8081, :3005	SearXNG, OpenWebUI up/down probes
frank-medium	192.168.0.235:7770	vLLM on Dankserver — 418 vllm: series
frank-large	192.168.0.143:7770	llama.cpp on Dankclaw — llamacpp: counters

The two LLM endpoints were added on 2026-09-16, after a night spent diagnosing frank-medium by hand. vLLM exports exactly what that diagnosis needed: vllm:kv_cache_usage_perc (how full the per-card pool is), vllm:num_requests_waiting (queue depth), the request-latency histograms, and the prefix-cache counters vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total, whose rates give the reuse rate that says whether a pinned conversation is being served warm or re-prefilled cold. What those numbers mean on this hardware is in frank-medium-radiance-mxfp4.

Both are scraped on their LAN addresses, never on the guarded vhosts: Prometheus cannot solve an Anubis challenge.

For services without a native exporter, use the Prometheus blackbox exporter in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.

Log aggregation choice

Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:

• Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
• Loki stores compressed logs with labels; queries run in Grafana's Explore.
• No Elasticsearch memory cost, no Kubernetes required.

Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain journald (no centralized query).

See Also

• Grafana, Prometheus, Loki, Alloy, Node-Exporter — the components
• Dankclaw Stack — the rest of the host
• Homepage — the user-facing status surface, including the five status cards
• Deployment Flow — the verify-before-declared-deployed contract