# Prometheus

Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints. It feeds the Grafana dashboards. The `node-exporter` containers may add podman/cgroup metrics for full host visibility.

**Host:** Dankclaw — **Hostname:** `prometheus.dankclaw.local` — **Port:** 9090 — **Docs:** https://prometheus.io/docs

## Facts

| Item | Value |
|------|-------|
| Image | `docker.io/prom/prometheus:latest` |
| Port | 9090 (host) → 9090 (container) |
| Config | `/etc/prometheus/prometheus.yml` (bind mount) |
| Data volume | TSDB |
| Retention | `--storage.tsdb.retention.time=30d` |

## prometheus.yml Targets

```yaml
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "comfyui"
    static_configs:
      - targets: ["comfyui:8188"]

  - job_name: "node"
    static_configs:
      - targets: ["node-exporter:9100"]
```

Allow the `/metrics` endpoint on each service where the exporter exists. For services without a native exporter, use the Prometheus `blackbox` exporter (probe mode) to check up/down and latency.

## Volumes

| Container path | Host path (convention) |
|----------------|------------------------|
| `/etc/prometheus` | `/opt/prometheus/config` |
| `/prometheus` | `/opt/prometheus/data` |

## Healthcheck

Prometheus responds on `/-/healthy` with 200.

## Gotchas

- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
- Use recording rules for heavy queries (see [[Grafana]]).
- Set a retention policy or the data volume grows without bound.

## See Also

- [[Dankclaw Stack]]
- [[Grafana]] — the dashboard frontend
- [[Observability]] — the whole stack