Frank
5829a0bfe8
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
54 lines
1.9 KiB
Markdown
54 lines
1.9 KiB
Markdown
# Observability
|
|
|
|
The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.
|
|
|
|
## Stack
|
|
|
|
```mermaid
|
|
graph LR
|
|
S[Services] -->|/metrics| P[Prometheus]
|
|
S -->|logs| T[Alloy]
|
|
T --> L[Loki]
|
|
P --> G[Grafana]
|
|
L --> G
|
|
```
|
|
|
|
| Component | Role | Image | Port |
|
|
|-----------|------|-------|------|
|
|
| [[Prometheus]] | metrics collection | `prom/prometheus` | 9090 |
|
|
| [[Grafana]] | dashboards | `grafana/grafana` | 3002 |
|
|
| Loki | log storage & query | `grafana/loki` | 3100 |
|
|
| Alloy | log shipping (per host) | `grafana/alloy` | — |
|
|
|
|
## Log Aggregation Choice
|
|
|
|
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
|
|
|
|
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
|
|
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
|
|
- No Elasticsearch memory cost, no Kubernetes required.
|
|
|
|
Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).
|
|
|
|
## Scrape Surface
|
|
|
|
| Target | Source |
|
|
|--------|--------|
|
|
| Prometheus self | `localhost:9090/metrics` |
|
|
| ComfyUI | `/system_stats` (limited) or blackbox probe |
|
|
| Gitea | `/metrics` (built-in exporter) |
|
|
| Grafana | `/metrics` (built-in) |
|
|
| Host metrics | node-exporter `:9100` |
|
|
|
|
For services without native exporters, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.
|
|
|
|
## Healthchecks
|
|
|
|
Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See [[Deployment Flow]] for the verify step.
|
|
|
|
## See Also
|
|
|
|
- [[Grafana]], [[Prometheus]]
|
|
- [[Dankclaw Stack]]
|
|
- [[Homepage]] — the user-facing status surface
|