Frank
e3eb5334cb
- Dankserver stack: RomM, Copyparty, Calibre-Web, node-exporter - Dankclaw stack: Gluetun VPN (SearXNG tunnel), SearXNG+Valkey, ComfyUI (CPU backend), OpenWebUI, Hermes, Gitea+Postgres (migrated), Caddy reverse proxy, Grafana+Prometheus+Loki+Alloy observability - Caddyfiles with .local hostname convention and basicauth homepage - bind-mount storage policy, no named volumes - deploy.sh mirroring vllmScripts git workflow (push -> pull -> restart -> health) - docs/ Obsidian vault: per-service docs from frank-default-small research subagents (promtail EOL -> Alloy; gitea SECRET_KEY preservation; ComfyUI --cpu) - bats test suite at 6 seams — 22 tests green
57 lines
1.7 KiB
Markdown
57 lines
1.7 KiB
Markdown
# Prometheus
|
|
|
|
Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints. It feeds the Grafana dashboards. The `node-exporter` containers may add podman/cgroup metrics for full host visibility.
|
|
|
|
**Host:** Dankclaw — **Hostname:** `prometheus.dankclaw.local` — **Port:** 9090 — **Docs:** https://prometheus.io/docs
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `docker.io/prom/prometheus:latest` |
|
|
| Port | 9090 (host) → 9090 (container) |
|
|
| Config | `/etc/prometheus/prometheus.yml` (bind mount) |
|
|
| Data volume | TSDB |
|
|
| Retention | `--storage.tsdb.retention.time=30d` |
|
|
|
|
## prometheus.yml Targets
|
|
|
|
```yaml
|
|
scrape_configs:
|
|
- job_name: "prometheus"
|
|
static_configs:
|
|
- targets: ["localhost:9090"]
|
|
|
|
- job_name: "comfyui"
|
|
static_configs:
|
|
- targets: ["comfyui:8188"]
|
|
|
|
- job_name: "node"
|
|
static_configs:
|
|
- targets: ["node-exporter:9100"]
|
|
```
|
|
|
|
Allow the `/metrics` endpoint on each service where the exporter exists. For services without a native exporter, use the Prometheus `blackbox` exporter (probe mode) to check up/down and latency.
|
|
|
|
## Volumes
|
|
|
|
| Container path | Host path (convention) |
|
|
|----------------|------------------------|
|
|
| `/etc/prometheus` | `/opt/prometheus/config` |
|
|
| `/prometheus` | `/opt/prometheus/data` |
|
|
|
|
## Healthcheck
|
|
|
|
Prometheus responds on `/-/healthy` with 200.
|
|
|
|
## Gotchas
|
|
|
|
- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
|
|
- Use recording rules for heavy queries (see [[Grafana]]).
|
|
- Set a retention policy or the data volume grows without bound.
|
|
|
|
## See Also
|
|
|
|
- [[Dankclaw Stack]]
|
|
- [[Grafana]] — the dashboard frontend
|
|
- [[Observability]] — the whole stack |