Fanned out 5 read-only doc-auditor subagents (one per service group) that fact-checked the knowledge base against the live system + repo config, researched official setup docs online, and verified progress. Applied: - All 34 broken links fixed (real pages / external URLs / correct relative paths); 0 broken links remain. No .AGENTTRASH links (the 4 mentions are accurate descriptive text, not links). - Fact-check fixes: Readarr retired (May 2026), Radarr v4->v6, Calibre 9.2.1 live, Calibre-Web 0.6.25 live, Copyparty v1.20.21, LLM index ncmoe 38 + 786432 ctx, frank-default-small active (not frank-pp-worker), Hermes/SearXNG/Grafana hostnames, Loki/Alloy local images, Observability diagram (Promtail->Alloy), VLLM retired note, DFlash status, benchmark live-drift notes, Homepage test count 116.
1.9 KiB
1.9 KiB
Observability
The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.
Stack
graph LR
S[Services] -->|/metrics| P[Prometheus]
S -->|logs| T[Alloy]
T --> L[Loki]
P --> G[Grafana]
L --> G
| Component | Role | Image | Port |
|---|---|---|---|
| Prometheus | metrics collection | prom/prometheus |
9090 |
| Grafana | dashboards | grafana/grafana |
3002 |
| Loki | log storage & query | grafana/loki |
3100 |
| Alloy | log shipping (per host) | grafana/alloy |
— |
Log Aggregation Choice
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
- No Elasticsearch memory cost, no Kubernetes required.
Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain journald (no centralized query).
Scrape Surface
| Target | Source |
|---|---|
| Prometheus self | localhost:9090/metrics |
| ComfyUI | /system_stats (limited) or blackbox probe |
| Gitea | /metrics (built-in exporter) |
| Grafana | /metrics (built-in) |
| Host metrics | node-exporter :9100 |
For services without native exporters, use the Prometheus blackbox exporter in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.
Healthchecks
Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See Deployment Flow for the verify step.
See Also
- Grafana, Prometheus
- Dankclaw Stack
- Homepage — the user-facing status surface