Frank
90c4ac69a7
Fanned out 5 read-only doc-auditor subagents (one per service group) that fact-checked the knowledge base against the live system + repo config, researched official setup docs online, and verified progress. Applied: - All 34 broken links fixed (real pages / external URLs / correct relative paths); 0 broken links remain. No .AGENTTRASH links (the 4 mentions are accurate descriptive text, not links). - Fact-check fixes: Readarr retired (May 2026), Radarr v4->v6, Calibre 9.2.1 live, Calibre-Web 0.6.25 live, Copyparty v1.20.21, LLM index ncmoe 38 + 786432 ctx, frank-default-small active (not frank-pp-worker), Hermes/SearXNG/Grafana hostnames, Loki/Alloy local images, Observability diagram (Promtail->Alloy), VLLM retired note, DFlash status, benchmark live-drift notes, Homepage test count 116.
54 lines
1.9 KiB
Markdown
54 lines
1.9 KiB
Markdown
# Observability
|
|
|
|
The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.
|
|
|
|
## Stack
|
|
|
|
```mermaid
|
|
graph LR
|
|
S[Services] -->|/metrics| P[Prometheus]
|
|
S -->|logs| T[Alloy]
|
|
T --> L[Loki]
|
|
P --> G[Grafana]
|
|
L --> G
|
|
```
|
|
|
|
| Component | Role | Image | Port |
|
|
|-----------|------|-------|------|
|
|
| [[Prometheus]] | metrics collection | `prom/prometheus` | 9090 |
|
|
| [[Grafana]] | dashboards | `grafana/grafana` | 3002 |
|
|
| Loki | log storage & query | `grafana/loki` | 3100 |
|
|
| Alloy | log shipping (per host) | `grafana/alloy` | — |
|
|
|
|
## Log Aggregation Choice
|
|
|
|
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
|
|
|
|
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
|
|
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
|
|
- No Elasticsearch memory cost, no Kubernetes required.
|
|
|
|
Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).
|
|
|
|
## Scrape Surface
|
|
|
|
| Target | Source |
|
|
|--------|--------|
|
|
| Prometheus self | `localhost:9090/metrics` |
|
|
| ComfyUI | `/system_stats` (limited) or blackbox probe |
|
|
| Gitea | `/metrics` (built-in exporter) |
|
|
| Grafana | `/metrics` (built-in) |
|
|
| Host metrics | node-exporter `:9100` |
|
|
|
|
For services without native exporters, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.
|
|
|
|
## Healthchecks
|
|
|
|
Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See [[Deployment Flow]] for the verify step.
|
|
|
|
## See Also
|
|
|
|
- [[Grafana]], [[Prometheus]]
|
|
- [[Dankclaw Stack]]
|
|
- [[Homepage]] — the user-facing status surface
|