serverAppConfigs/serverAppConfigs-docs/Infrastructure/Observability.md

---
type: reference
title: Observability on Dankclaw
description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy and node-exporter: what runs, how it is deployed, how it is exposed, and what is still unconfigured.
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
timestamp: 2026-09-11
---

# Observability

The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards and node-exporter for host metrics. This is a small homelab — no Kubernetes.

## Stack

```mermaid
graph LR
    S[Services] -->|/metrics| P[Prometheus]
    H[Host] -->|":9100"| P
    S -->|logs| A[Alloy]
    J[journald] --> A
    A --> L[Loki]
    P --> G[Grafana]
    L --> G
```

| Component | Role | Image | Published |
|-----------|------|-------|-----------|
| [[Prometheus]] | metrics collection | `prom/prometheus` | `127.0.0.1:9090` |
| [[Grafana]] | dashboards | `grafana/grafana` | `127.0.0.1:3002` |
| [[Loki]] | log storage & query | `franksdank/loki` (local build) | `127.0.0.1:3100` |
| [[Alloy]] | log shipping | `franksdank/alloy` (local build) | `127.0.0.1:12345` |
| [[Node-Exporter]] | host metrics | `prom/node-exporter` | `192.168.0.143:9100`, host netns |

## Current state: installed, healthy, and mostly unconfigured

Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:

- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `vllm:kv_cache_usage_perc` for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
- **Prometheus has no alerting at all.** `/opt/prometheus/config/` contains `prometheus.yml` and nothing else — no rule files, no recording rules — and there is no Alertmanager anywhere in the stack. It scrapes and retains; it never notifies anyone.
- **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
- **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
- **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.

The honest summary: metrics and logs are collected and retained, one dashboard now visualizes the LLM fleet, and **nothing is alerted on** — there is no rules file and no Alertmanager anywhere, so every graph in this stack is read after the fact by a human, and correlating metrics with logs is still manual.

## Deployment

All five are rootless **Quadlet units** on Dankclaw. The units live in `dankclaw/observability/quadlet/<name>.container` and install to `~/.config/containers/systemd/`, generating `<name>.service`.

They left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots.

```bash
./scripts/deploy.sh dankclaw --service loki        # or prometheus|grafana|alloy|node-exporter
```

`deploy_observability` in `scripts/deploy-remote.sh` installs the unit, restarts it and waits on its endpoint. Ordering lives in the units themselves (Grafana wants Prometheus, Alloy wants Loki), and `SERVICE_ORDER` in the planner puts each dependency first.

Every unit carries a healthcheck, so a wedged process reads `unhealthy` in `podman ps` rather than merely `Up`. See [[Loki]] for the one unit that cannot declare it inline.

## Exposure

The rule is the same as for every other service: the only off-host path is a guarded vhost (`<name>.franksdank.cloud` → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong:

- **Prometheus, Grafana, Loki and Alloy publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
- **node-exporter is the one deliberate LAN listener.** Prometheus runs in a container and cannot reach the host's loopback, so the unit binds exactly two addresses: `127.0.0.1:9100` for the homepage status probe and `192.168.0.143:9100` for Prometheus. It authenticates nothing, so it never gets a vhost.

## Scrape surface

`shared/prometheus/prometheus.yml` is the source of truth (installed to `/opt/prometheus/config/` on every deploy). It scrapes:

| Job | Target | Notes |
|-----|--------|-------|
| `prometheus` | `localhost:9090` | itself |
| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | one node-exporter per host |
| `gitea` | `192.168.0.143:3000` | published LAN port (Gitea is on an isolated network) |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` |
| `comfyui` | `comfyui:8188` | UI probe, no native exporter |
| `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes |
| `frank-medium` | `192.168.0.235:7770` | vLLM on Dankserver — 418 `vllm:` series |
| `frank-large` | `192.168.0.143:7770` | llama.cpp on Dankclaw — `llamacpp:` counters |

The two LLM endpoints were added on 2026-09-16, after a night spent diagnosing
frank-medium by hand. vLLM exports exactly what that diagnosis needed:
`vllm:kv_cache_usage_perc` (how full the per-card pool is), 
`vllm:num_requests_waiting` (queue depth), the request-latency histograms, and
the prefix-cache counters `vllm:prefix_cache_hits_total` /
`vllm:prefix_cache_queries_total`, whose rates give the reuse rate that says
whether a pinned conversation is being served warm or re-prefilled cold. What
those numbers mean on this hardware is in [[frank-medium-radiance-mxfp4]].

Both are scraped on their LAN addresses, never on the guarded vhosts:
Prometheus cannot solve an Anubis challenge.

For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.

## Log aggregation choice

Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:

- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
- No Elasticsearch memory cost, no Kubernetes required.

Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).

## See Also

- [[Grafana]], [[Prometheus]], [[Loki]], [[Alloy]], [[Node-Exporter]] — the components
- [[Dankclaw Stack]] — the rest of the host
- [[Homepage]] — the user-facing status surface, including the five status cards
- [[Deployment Flow]] — the verify-before-declared-deployed contract