Frank
e3eb5334cb
- Dankserver stack: RomM, Copyparty, Calibre-Web, node-exporter - Dankclaw stack: Gluetun VPN (SearXNG tunnel), SearXNG+Valkey, ComfyUI (CPU backend), OpenWebUI, Hermes, Gitea+Postgres (migrated), Caddy reverse proxy, Grafana+Prometheus+Loki+Alloy observability - Caddyfiles with .local hostname convention and basicauth homepage - bind-mount storage policy, no named volumes - deploy.sh mirroring vllmScripts git workflow (push -> pull -> restart -> health) - docs/ Obsidian vault: per-service docs from frank-default-small research subagents (promtail EOL -> Alloy; gitea SECRET_KEY preservation; ComfyUI --cpu) - bats test suite at 6 seams — 22 tests green
54 lines
1.9 KiB
Markdown
54 lines
1.9 KiB
Markdown
# Observability
|
|
|
|
The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.
|
|
|
|
## Stack
|
|
|
|
```mermaid
|
|
graph LR
|
|
S[Services] -->|/metrics| P[Prometheus]
|
|
S -->|logs| T[Promtail]
|
|
T --> L[Loki]
|
|
P --> G[Grafana]
|
|
L --> G
|
|
```
|
|
|
|
| Component | Role | Image | Port |
|
|
|-----------|------|-------|------|
|
|
| [[Prometheus]] | metrics collection | `prom/prometheus` | 9090 |
|
|
| [[Grafana]] | dashboards | `grafana/grafana` | 3002 |
|
|
| Loki | log storage & query | `grafana/loki` | 3100 |
|
|
| Alloy | log shipping (per host) | `grafana/alloy` | — |
|
|
|
|
## Log Aggregation Choice
|
|
|
|
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
|
|
|
|
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
|
|
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
|
|
- No Elasticsearch memory cost, no Kubernetes required.
|
|
|
|
Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).
|
|
|
|
## Scrape Surface
|
|
|
|
| Target | Source |
|
|
|--------|--------|
|
|
| Prometheus self | `localhost:9090/metrics` |
|
|
| ComfyUI | `/system_stats` (limited) or blackbox probe |
|
|
| Gitea | `/metrics` (built-in exporter) |
|
|
| Grafana | `/metrics` (built-in) |
|
|
| Host metrics | node-exporter `:9100` |
|
|
|
|
For services without native exporters, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.
|
|
|
|
## Healthchecks
|
|
|
|
Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See [[Deployment Flow]] for the verify step.
|
|
|
|
## See Also
|
|
|
|
- [[Grafana]], [[Prometheus]]
|
|
- [[Dankclaw Stack]]
|
|
- [[Homepage]] — the user-facing status surface
|