Frank
823814920b
Both pages claimed there was no alerting; there are six rules now, so they say which rules exist, that promtool and /api/v1/rules both confirm them, and — in the same breath — that no rule reaches a person, because this stack has no Alertmanager and no contact point. The dashboard's ALERTS panel is named as the only place an alert becomes visible without opening Prometheus.
113 lines
8.7 KiB
Markdown
113 lines
8.7 KiB
Markdown
---
|
|
type: reference
|
|
title: Observability on Dankclaw
|
|
description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy and node-exporter: what runs, how it is deployed, how it is exposed, and what is still unconfigured.
|
|
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
|
|
timestamp: 2026-09-11
|
|
---
|
|
|
|
# Observability
|
|
|
|
The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards and node-exporter for host metrics. This is a small homelab — no Kubernetes.
|
|
|
|
## Stack
|
|
|
|
```mermaid
|
|
graph LR
|
|
S[Services] -->|/metrics| P[Prometheus]
|
|
H[Host] -->|":9100"| P
|
|
S -->|logs| A[Alloy]
|
|
J[journald] --> A
|
|
A --> L[Loki]
|
|
P --> G[Grafana]
|
|
L --> G
|
|
```
|
|
|
|
| Component | Role | Image | Published |
|
|
|-----------|------|-------|-----------|
|
|
| [[Prometheus]] | metrics collection | `prom/prometheus` | `127.0.0.1:9090` |
|
|
| [[Grafana]] | dashboards | `grafana/grafana` | `127.0.0.1:3002` |
|
|
| [[Loki]] | log storage & query | `franksdank/loki` (local build) | `127.0.0.1:3100` |
|
|
| [[Alloy]] | log shipping | `franksdank/alloy` (local build) | `127.0.0.1:12345` |
|
|
| [[Node-Exporter]] | host metrics | `prom/node-exporter` | `192.168.0.143:9100`, host netns |
|
|
|
|
## Current state: installed, healthy, and mostly unconfigured
|
|
|
|
Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:
|
|
|
|
- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `vllm:kv_cache_usage_perc` for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
|
|
- **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
|
|
- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds six rules for the LLM fleet: pool saturation, request queueing, prefix-cache collapse, endpoint down, p95 latency, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. Verified with `promtool check rules` (SUCCESS, 6 rules) and `/api/v1/rules` (all six `health=ok`). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard's alerts panel — and reaches a human only if one is looking.
|
|
- **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
|
|
- **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
|
|
- **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
|
|
|
|
The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, six rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at.
|
|
|
|
## Deployment
|
|
|
|
All five are rootless **Quadlet units** on Dankclaw. The units live in `dankclaw/observability/quadlet/<name>.container` and install to `~/.config/containers/systemd/`, generating `<name>.service`.
|
|
|
|
They left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots.
|
|
|
|
```bash
|
|
./scripts/deploy.sh dankclaw --service loki # or prometheus|grafana|alloy|node-exporter
|
|
```
|
|
|
|
`deploy_observability` in `scripts/deploy-remote.sh` installs the unit, restarts it and waits on its endpoint. Ordering lives in the units themselves (Grafana wants Prometheus, Alloy wants Loki), and `SERVICE_ORDER` in the planner puts each dependency first.
|
|
|
|
Every unit carries a healthcheck, so a wedged process reads `unhealthy` in `podman ps` rather than merely `Up`. See [[Loki]] for the one unit that cannot declare it inline.
|
|
|
|
## Exposure
|
|
|
|
The rule is the same as for every other service: the only off-host path is a guarded vhost (`<name>.franksdank.cloud` → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong:
|
|
|
|
- **Prometheus, Grafana, Loki and Alloy publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
|
|
- **node-exporter is the one deliberate LAN listener.** Prometheus runs in a container and cannot reach the host's loopback, so the unit binds exactly two addresses: `127.0.0.1:9100` for the homepage status probe and `192.168.0.143:9100` for Prometheus. It authenticates nothing, so it never gets a vhost.
|
|
|
|
## Scrape surface
|
|
|
|
`shared/prometheus/prometheus.yml` is the source of truth (installed to `/opt/prometheus/config/` on every deploy). It scrapes:
|
|
|
|
| Job | Target | Notes |
|
|
|-----|--------|-------|
|
|
| `prometheus` | `localhost:9090` | itself |
|
|
| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | one node-exporter per host |
|
|
| `gitea` | `192.168.0.143:3000` | published LAN port (Gitea is on an isolated network) |
|
|
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` |
|
|
| `comfyui` | `comfyui:8188` | UI probe, no native exporter |
|
|
| `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes |
|
|
| `frank-medium` | `192.168.0.235:7770` | vLLM on Dankserver — 418 `vllm:` series |
|
|
| `frank-large` | `192.168.0.143:7770` | llama.cpp on Dankclaw — `llamacpp:` counters |
|
|
|
|
The two LLM endpoints were added on 2026-09-16, after a night spent diagnosing
|
|
frank-medium by hand. vLLM exports exactly what that diagnosis needed:
|
|
`vllm:kv_cache_usage_perc` (how full the per-card pool is),
|
|
`vllm:num_requests_waiting` (queue depth), the request-latency histograms, and
|
|
the prefix-cache counters `vllm:prefix_cache_hits_total` /
|
|
`vllm:prefix_cache_queries_total`, whose rates give the reuse rate that says
|
|
whether a pinned conversation is being served warm or re-prefilled cold. What
|
|
those numbers mean on this hardware is in [[frank-medium-radiance-mxfp4]].
|
|
|
|
Both are scraped on their LAN addresses, never on the guarded vhosts:
|
|
Prometheus cannot solve an Anubis challenge.
|
|
|
|
For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.
|
|
|
|
## Log aggregation choice
|
|
|
|
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
|
|
|
|
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
|
|
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
|
|
- No Elasticsearch memory cost, no Kubernetes required.
|
|
|
|
Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).
|
|
|
|
## See Also
|
|
|
|
- [[Grafana]], [[Prometheus]], [[Loki]], [[Alloy]], [[Node-Exporter]] — the components
|
|
- [[Dankclaw Stack]] — the rest of the host
|
|
- [[Homepage]] — the user-facing status surface, including the five status cards
|
|
- [[Deployment Flow]] — the verify-before-declared-deployed contract
|