|
|
|
@@ -1,151 +1,7 @@
|
|
|
|
|
---
|
|
|
|
|
type: reference
|
|
|
|
|
title: Observability on Dankclaw
|
|
|
|
|
description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured.
|
|
|
|
|
title: "Observability on Dankclaw"
|
|
|
|
|
description: "The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured."
|
|
|
|
|
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
|
|
|
|
|
timestamp: 2026-09-11
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# Observability
|
|
|
|
|
|
|
|
|
|
The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards, node-exporter for host metrics, and Alertmanager for alert delivery — to Signal, through a host-side relay. This is a small homelab — no Kubernetes.
|
|
|
|
|
|
|
|
|
|
## Stack
|
|
|
|
|
|
|
|
|
|
```mermaid
|
|
|
|
|
graph LR
|
|
|
|
|
S[Services] -->|/metrics| P[Prometheus]
|
|
|
|
|
H[Host] -->|":9100"| P
|
|
|
|
|
S -->|logs| A[Alloy]
|
|
|
|
|
J[journald] --> A
|
|
|
|
|
A --> L[Loki]
|
|
|
|
|
P --> G[Grafana]
|
|
|
|
|
L --> G
|
|
|
|
|
P -->|alerts| AM[Alertmanager]
|
|
|
|
|
AM -->|webhook| R[Signal relay]
|
|
|
|
|
R -->|signal-cli| SIG[Signal]
|
|
|
|
|
|
|
|
|
|
| Component | Role | Image | Published |
|
|
|
|
|
|-----------|------|-------|-----------|
|
|
|
|
|
| [[Prometheus]] | metrics collection | `prom/prometheus` | `127.0.0.1:9090` |
|
|
|
|
|
| [[Grafana]] | dashboards | `grafana/grafana` | `127.0.0.1:3002` |
|
|
|
|
|
| [[Loki]] | log storage & query | `franksdank/loki` (local build) | `127.0.0.1:3100` |
|
|
|
|
|
| [[Alloy]] | log shipping | `franksdank/alloy` (local build) | `127.0.0.1:12345` |
|
|
|
|
|
| [[Node-Exporter]] | host metrics | `prom/node-exporter` | `192.168.0.143:9100`, host netns |
|
|
|
|
|
| [[Alertmanager]] | alert grouping & delivery | `prom/alertmanager` | `127.0.0.1:9093` |
|
|
|
|
|
|
|
|
|
|
## Current state: installed, healthy, and mostly unconfigured
|
|
|
|
|
|
|
|
|
|
Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:
|
|
|
|
|
|
|
|
|
|
- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
|
|
|
|
|
- **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
|
|
|
|
|
- **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the relay binds the host's LAN IP (192.168.0.143:18093, the one address netavark delivers that dial to), with an iptables set admitting loopback, the host and the podman bridge subnet only. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
|
|
|
|
|
- **2026-09-20: the rule file now spans two metric namespaces, and the dashboard was re-cut the same day.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — six alerts and three recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. One semantics trap, measured 2026-09-20: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP line), so the llama.cpp reuse rule divides by `cached + prompt` — the naive `cached/prompt` ratio reads 77 on a fully-warm endpoint and can never show a collapse. The provisioned dashboard's panels now read both namespaces (the endpoint picker selects between them); the KV-pool-fill panel retired 2026-09-16 returned with `vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every expression in the rules and the dashboard was validated against the live endpoints on 2026-09-20.
|
|
|
|
|
- **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
|
|
|
|
|
- **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
|
|
|
|
|
- **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
|
|
|
|
|
|
|
|
|
|
The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, nine rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent.
|
|
|
|
|
|
|
|
|
|
## Deployment
|
|
|
|
|
|
|
|
|
|
Seven units now, across two hosts. Dankclaw runs six (Prometheus, Grafana, Loki,
|
|
|
|
|
Alloy, node-exporter, Alertmanager) from `dankclaw/observability/quadlet/`; **Dankserver runs
|
|
|
|
|
its own node-exporter** from `dankserver/observability/quadlet/`, added
|
|
|
|
|
2026-09-16 after the same reboot argument took it down a second time — it had
|
|
|
|
|
been a compose-only service, died `Exited (143)` at a reboot, and left
|
|
|
|
|
Prometheus' `node-dankserver` job scraping nothing for a day and a half (its
|
|
|
|
|
compose form also bound the wildcard address). The deployer installs from
|
|
|
|
|
`"$TARGET/observability/quadlet"`, so each host owns its own units.
|
|
|
|
|
|
|
|
|
|
The Dankclaw five left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots, and the one that finally caught Dankserver's node-exporter. **What runs is now the unit; the compose file is the historical record.**
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
./scripts/deploy.sh dankclaw --service loki # or prometheus|grafana|alloy|node-exporter|alertmanager
|
|
|
|
|
./scripts/deploy.sh dankserver --service node-exporter
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`deploy_observability` in `scripts/deploy-remote.sh` installs the unit, restarts it and waits on its endpoint. Ordering lives in the units themselves (Grafana wants Prometheus, Alloy wants Loki), and `SERVICE_ORDER` in the planner puts each dependency first.
|
|
|
|
|
|
|
|
|
|
Every unit carries a healthcheck, so a wedged process reads `unhealthy` in `podman ps` rather than merely `Up`. See [[Loki]] for the one unit that cannot declare it inline.
|
|
|
|
|
|
|
|
|
|
## Exposure
|
|
|
|
|
|
|
|
|
|
The rule is the same as for every other service: the only off-host path is a guarded vhost (`<name>.franksdank.cloud` → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong:
|
|
|
|
|
|
|
|
|
|
- **Prometheus, Grafana, Loki, Alloy and Alertmanager publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
|
|
|
|
|
- **node-exporter is the one deliberate LAN listener, and there are two of them now — one per host.** Prometheus runs in a container and cannot reach either host's loopback, so each unit binds exactly two addresses: loopback (`127.0.0.1:9100`) for host-local probes and its own LAN address (`192.168.0.143:9100` on Dankclaw, `192.168.0.235:9100` on Dankserver) for Prometheus. `ss -ltn` on Dankserver shows exactly those two listeners and no wildcard. Neither authenticates anything, so neither gets a vhost.
|
|
|
|
|
|
|
|
|
|
## Scrape surface
|
|
|
|
|
|
|
|
|
|
`shared/prometheus/prometheus.yml` is the source of truth (installed to `/opt/prometheus/config/` on every deploy). It scrapes:
|
|
|
|
|
|
|
|
|
|
| Job | Target | Notes |
|
|
|
|
|
|-----|--------|-------|
|
|
|
|
|
| `prometheus` | `localhost:9090` | itself |
|
|
|
|
|
| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | one node-exporter per host |
|
|
|
|
|
| `gitea` | `192.168.0.143:3000` | published LAN port (Gitea is on an isolated network) |
|
|
|
|
|
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` |
|
|
|
|
|
| `comfyui` | `comfyui:8188` | UI probe, no native exporter |
|
|
|
|
|
| `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes |
|
|
|
|
|
| `frank-medium` | `192.168.0.235:7770` | radiance/vLLM on Dankserver — `vllm:` counters: KV fill, queue depth, preemptions, prefix-cache reuse, and the `spec_decode_*` trio |
|
|
|
|
|
| `frank-small` | `host.containers.internal:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair |
|
|
|
|
|
| `frank-large` | `host.containers.internal:7770` | CIRU llama.cpp on Dankclaw's iGPU — 19 `llamacpp:` families |
|
|
|
|
|
|
|
|
|
|
The LLM endpoints were added on 2026-09-16, after a night spent diagnosing
|
|
|
|
|
frank-medium by hand, and re-cut the same day when Dankclaw split into two
|
|
|
|
|
endpoints. frank-medium has since changed runtime twice (vLLM → mainline
|
|
|
|
|
llama.cpp → radiance/vLLM on 2026-09-20), and the metric family it exports
|
|
|
|
|
changed with it: `llamacpp:*` while it ran llama.cpp, `vllm:*` now.
|
|
|
|
|
|
|
|
|
|
**Both Dankclaw targets are addressed as `host.containers.internal`
|
|
|
|
|
(169.254.1.2), never the LAN IP.** frank-large and frank-small are *host
|
|
|
|
|
processes* on Dankclaw, and netavark DNATs only *published* container ports —
|
|
|
|
|
from the Prometheus container `192.168.0.143:7770/:7771` is refused even
|
|
|
|
|
though it answers from every other vantage on the LAN. Both targets were red
|
|
|
|
|
by construction until a live `/targets` audit caught it on 2026-09-20.
|
|
|
|
|
`frank-medium` keeps its LAN address: it is a rootless Quadlet on another host
|
|
|
|
|
that publishes 7770, so its published-port DNAT does apply.
|
|
|
|
|
|
|
|
|
|
**`frank-large` IS scraped as of 2026-09-20.** The earlier claim that it had
|
|
|
|
|
"no Prometheus text endpoint at all" was true of the halogen engine it ran
|
|
|
|
|
then; the CIRU llama.cpp build that replaced it exports the full
|
|
|
|
|
`llamacpp:*` family, which is what makes the target addressable at all. Its
|
|
|
|
|
liveness is additionally on the homepage (a loopback probe) and in the
|
|
|
|
|
engine's own boot ledger.
|
|
|
|
|
|
|
|
|
|
For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.
|
|
|
|
|
|
|
|
|
|
**Four targets read "down" — three jobs, and every one of them is structural**
|
|
|
|
|
(found 2026-09-16 while auditing why the target list was not green):
|
|
|
|
|
|
|
|
|
|
| job | error | what it means |
|
|
|
|
|
|-----|-------|---------------|
|
|
|
|
|
| `comfyui` | `unsupported Content-Type "text/html"` | the service is **up** (this is what it says since ComfyUI was fixed — before that it read `connection refused`); Prometheus refuses to parse an HTML page as metrics, so this job cannot succeed as configured |
|
|
|
|
|
| `services` (SearXNG, OpenWebUI) | `unsupported Content-Type "text/html; charset=utf-8"` | same defect: "blackbox-style probes" that scrape a page instead of metrics |
|
|
|
|
|
| `gitea` | `HTTP status 404 Not Found` | `/metrics` does not exist on that Gitea (metrics disabled); the service itself is fine |
|
|
|
|
|
|
|
|
|
|
None of these is an outage, and none should be silenced by deleting the job —
|
|
|
|
|
an absent scrape and a healthy one look identical. The fix the repository
|
|
|
|
|
already names is a **blackbox exporter** in probe mode for the page-only
|
|
|
|
|
targets, and either enabling Gitea's metrics or pointing the job at the right
|
|
|
|
|
path. The LLM alert rules (`LlmEndpointDown`) are scoped to `frank-*`, so these
|
|
|
|
|
probes generate no false pages.
|
|
|
|
|
|
|
|
|
|
## Log aggregation choice
|
|
|
|
|
|
|
|
|
|
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
|
|
|
|
|
|
|
|
|
|
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
|
|
|
|
|
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
|
|
|
|
|
- No Elasticsearch memory cost, no Kubernetes required.
|
|
|
|
|
|
|
|
|
|
Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).
|
|
|
|
|
|
|
|
|
|
## See Also
|
|
|
|
|
|
|
|
|
|
- [[Grafana]], [[Prometheus]], [[Loki]], [[Alloy]], [[Node-Exporter]] — the components
|
|
|
|
|
- [[Dankclaw Stack]] — the rest of the host
|
|
|
|
|
- [[Homepage]] — the user-facing status surface, including the five status cards
|
|
|
|
|
- [[Deployment Flow]] — the verify-before-declared-deployed contract
|
|
|
|
|
Frank