Files
serverAppConfigs/docs/Infrastructure/Observability.md
T
Frank 90c4ac69a7 Doc audit: fix all broken links + correct stale facts across the KB
Fanned out 5 read-only doc-auditor subagents (one per service group) that
fact-checked the knowledge base against the live system + repo config,
researched official setup docs online, and verified progress. Applied:
- All 34 broken links fixed (real pages / external URLs / correct relative
  paths); 0 broken links remain. No .AGENTTRASH links (the 4 mentions are
  accurate descriptive text, not links).
- Fact-check fixes: Readarr retired (May 2026), Radarr v4->v6, Calibre
  9.2.1 live, Calibre-Web 0.6.25 live, Copyparty v1.20.21, LLM index
  ncmoe 38 + 786432 ctx, frank-default-small active (not frank-pp-worker),
  Hermes/SearXNG/Grafana hostnames, Loki/Alloy local images, Observability
  diagram (Promtail->Alloy), VLLM retired note, DFlash status, benchmark
  live-drift notes, Homepage test count 116.
2026-08-24 01:10:44 -07:00

1.9 KiB

Observability

The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.

Stack

graph LR
    S[Services] -->|/metrics| P[Prometheus]
    S -->|logs| T[Alloy]
    T --> L[Loki]
    P --> G[Grafana]
    L --> G
Component Role Image Port
Prometheus metrics collection prom/prometheus 9090
Grafana dashboards grafana/grafana 3002
Loki log storage & query grafana/loki 3100
Alloy log shipping (per host) grafana/alloy —

Log Aggregation Choice

Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:

  • Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
  • Loki stores compressed logs with labels; queries run in Grafana's Explore.
  • No Elasticsearch memory cost, no Kubernetes required.

Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain journald (no centralized query).

Scrape Surface

Target Source
Prometheus self localhost:9090/metrics
ComfyUI /system_stats (limited) or blackbox probe
Gitea /metrics (built-in exporter)
Grafana /metrics (built-in)
Host metrics node-exporter :9100

For services without native exporters, use the Prometheus blackbox exporter in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.

Healthchecks

Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See Deployment Flow for the verify step.

See Also