# Observability

The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.

## Stack

```mermaid
graph LR
    S[Services] -->|/metrics| P[Prometheus]
    S -->|logs| T[Alloy]
    T --> L[Loki]
    P --> G[Grafana]
    L --> G
```

| Component | Role | Image | Port |
|-----------|------|-------|------|
| [[Prometheus]] | metrics collection | `prom/prometheus` | 9090 |
| [[Grafana]] | dashboards | `grafana/grafana` | 3002 |
| Loki | log storage & query | `grafana/loki` | 3100 |
| Alloy | log shipping (per host) | `grafana/alloy` | — |

## Log Aggregation Choice

Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:

- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
- No Elasticsearch memory cost, no Kubernetes required.

Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).

## Scrape Surface

| Target | Source |
|--------|--------|
| Prometheus self | `localhost:9090/metrics` |
| ComfyUI | `/system_stats` (limited) or blackbox probe |
| Gitea | `/metrics` (built-in exporter) |
| Grafana | `/metrics` (built-in) |
| Host metrics | node-exporter `:9100` |

For services without native exporters, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.

## Healthchecks

Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See [[Deployment Flow]] for the verify step.

## See Also

- [[Grafana]], [[Prometheus]]
- [[Dankclaw Stack]]
- [[Homepage]] — the user-facing status surface