Files

T

Frank c42780f9f3 fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity

- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
  divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
  cached tokens (its own HELP line), so the ratio was unbounded — measured
  77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
  fire on that side. Now cached / (cached + prompt), with the traffic floor
  counting both terms too; the header documents the unit trap. Also fixed the
  LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
  the llamacpp:* family (verified live), so the alert covers all three
  endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
  namespaces. Every panel reads both (the endpoint picker selects); the
  KV-pool-fill panel retired 2026-09-16 returned with
  vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
  histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
  Llm.* rules; the stale halogen-era description is gone. All 21 panel
  expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
  reuse happens at 1,648-token block granularity (attention block size padded
  to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
  to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
  registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.

./tests/run.sh: 325 pass.

2026-09-20 19:53:14 -07:00

15 KiB

Raw Blame History

type: reference title: Observability on Dankclaw description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured. tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw] timestamp: 2026-09-11

Observability

The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards, node-exporter for host metrics, and Alertmanager for alert delivery — to Signal, through a host-side relay. This is a small homelab — no Kubernetes.

Stack

graph LR
    S[Services] -->|/metrics| P[Prometheus]
    H[Host] -->|":9100"| P
    S -->|logs| A[Alloy]
    J[journald] --> A
    A --> L[Loki]
    P --> G[Grafana]
    L --> G
    P -->|alerts| AM[Alertmanager]
    AM -->|webhook| R[Signal relay]
    R -->|signal-cli| SIG[Signal]

| Component | Role | Image | Published |
|-----------|------|-------|-----------|
| [[Prometheus]] | metrics collection | `prom/prometheus` | `127.0.0.1:9090` |
| [[Grafana]] | dashboards | `grafana/grafana` | `127.0.0.1:3002` |
| [[Loki]] | log storage & query | `franksdank/loki` (local build) | `127.0.0.1:3100` |
| [[Alloy]] | log shipping | `franksdank/alloy` (local build) | `127.0.0.1:12345` |
| [[Node-Exporter]] | host metrics | `prom/node-exporter` | `192.168.0.143:9100`, host netns |
| [[Alertmanager]] | alert grouping & delivery | `prom/alertmanager` | `127.0.0.1:9093` |

## Current state: installed, healthy, and mostly unconfigured

Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:

- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
- **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
- **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the relay binds the host's LAN IP (192.168.0.143:18093, the one address netavark delivers that dial to), with an iptables set admitting loopback, the host and the podman bridge subnet only. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
- **2026-09-20: the rule file now spans two metric namespaces, and the dashboard was re-cut the same day.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — six alerts and three recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. One semantics trap, measured 2026-09-20: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP line), so the llama.cpp reuse rule divides by `cached + prompt` — the naive `cached/prompt` ratio reads 77 on a fully-warm endpoint and can never show a collapse. The provisioned dashboard's panels now read both namespaces (the endpoint picker selects between them); the KV-pool-fill panel retired 2026-09-16 returned with `vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every expression in the rules and the dashboard was validated against the live endpoints on 2026-09-20.
- **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
- **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
- **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.

The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, nine rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent.

## Deployment

Seven units now, across two hosts. Dankclaw runs six (Prometheus, Grafana, Loki,
Alloy, node-exporter, Alertmanager) from `dankclaw/observability/quadlet/`; **Dankserver runs
its own node-exporter** from `dankserver/observability/quadlet/`, added
2026-09-16 after the same reboot argument took it down a second time — it had
been a compose-only service, died `Exited (143)` at a reboot, and left
Prometheus' `node-dankserver` job scraping nothing for a day and a half (its
compose form also bound the wildcard address). The deployer installs from
`"$TARGET/observability/quadlet"`, so each host owns its own units.

The Dankclaw five left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots, and the one that finally caught Dankserver's node-exporter. **What runs is now the unit; the compose file is the historical record.**

```bash
./scripts/deploy.sh dankclaw --service loki          # or prometheus|grafana|alloy|node-exporter|alertmanager
./scripts/deploy.sh dankserver --service node-exporter

deploy_observability in scripts/deploy-remote.sh installs the unit, restarts it and waits on its endpoint. Ordering lives in the units themselves (Grafana wants Prometheus, Alloy wants Loki), and SERVICE_ORDER in the planner puts each dependency first.

Every unit carries a healthcheck, so a wedged process reads unhealthy in podman ps rather than merely Up. See Loki for the one unit that cannot declare it inline.

Exposure

The rule is the same as for every other service: the only off-host path is a guarded vhost (<name>.franksdank.cloud → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong:

Prometheus, Grafana, Loki, Alloy and Alertmanager publish on 127.0.0.1 only. Caddy reaches them by container name over the dankclaw_default network, and the host-side homepage status timer probes loopback. Publishing on 0.0.0.0 — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
node-exporter is the one deliberate LAN listener, and there are two of them now — one per host. Prometheus runs in a container and cannot reach either host's loopback, so each unit binds exactly two addresses: loopback (127.0.0.1:9100) for host-local probes and its own LAN address (192.168.0.143:9100 on Dankclaw, 192.168.0.235:9100 on Dankserver) for Prometheus. ss -ltn on Dankserver shows exactly those two listeners and no wildcard. Neither authenticates anything, so neither gets a vhost.

Scrape surface

shared/prometheus/prometheus.yml is the source of truth (installed to /opt/prometheus/config/ on every deploy). It scrapes:

Job	Target	Notes
`prometheus`	`localhost:9090`	itself
`node-dankclaw`, `node-dankserver`	`192.168.0.143:9100`, `192.168.0.235:9100`	one node-exporter per host
`gitea`	`192.168.0.143:3000`	published LAN port (Gitea is on an isolated network)
`grafana`, `loki`	`grafana:3000`, `loki:3100`	container names on `dankclaw_default`
`comfyui`	`comfyui:8188`	UI probe, no native exporter
`services`	`192.168.0.143:8081`, `:3005`	SearXNG, OpenWebUI up/down probes
`frank-medium`	`192.168.0.235:7770`	radiance/vLLM on Dankserver — `vllm:` counters: KV fill, queue depth, preemptions, prefix-cache reuse, and the `spec_decode_*` trio
`frank-small`	`host.containers.internal:7771`	llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair
`frank-large`	`host.containers.internal:7770`	CIRU llama.cpp on Dankclaw's iGPU — 19 `llamacpp:` families

The LLM endpoints were added on 2026-09-16, after a night spent diagnosing frank-medium by hand, and re-cut the same day when Dankclaw split into two endpoints. frank-medium has since changed runtime twice (vLLM → mainline llama.cpp → radiance/vLLM on 2026-09-20), and the metric family it exports changed with it: llamacpp:* while it ran llama.cpp, vllm:* now.

Both Dankclaw targets are addressed as host.containers.internal (169.254.1.2), never the LAN IP. frank-large and frank-small are host processes on Dankclaw, and netavark DNATs only published container ports — from the Prometheus container 192.168.0.143:7770/:7771 is refused even though it answers from every other vantage on the LAN. Both targets were red by construction until a live /targets audit caught it on 2026-09-20. frank-medium keeps its LAN address: it is a rootless Quadlet on another host that publishes 7770, so its published-port DNAT does apply.

frank-large IS scraped as of 2026-09-20. The earlier claim that it had "no Prometheus text endpoint at all" was true of the halogen engine it ran then; the CIRU llama.cpp build that replaced it exports the full llamacpp:* family, which is what makes the target addressable at all. Its liveness is additionally on the homepage (a loopback probe) and in the engine's own boot ledger.

For services without a native exporter, use the Prometheus blackbox exporter in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.

Four targets read "down" — three jobs, and every one of them is structural (found 2026-09-16 while auditing why the target list was not green):

job	error	what it means
`comfyui`	`unsupported Content-Type "text/html"`	the service is up (this is what it says since ComfyUI was fixed — before that it read `connection refused`); Prometheus refuses to parse an HTML page as metrics, so this job cannot succeed as configured
`services` (SearXNG, OpenWebUI)	`unsupported Content-Type "text/html; charset=utf-8"`	same defect: "blackbox-style probes" that scrape a page instead of metrics
`gitea`	`HTTP status 404 Not Found`	`/metrics` does not exist on that Gitea (metrics disabled); the service itself is fine

None of these is an outage, and none should be silenced by deleting the job — an absent scrape and a healthy one look identical. The fix the repository already names is a blackbox exporter in probe mode for the page-only targets, and either enabling Gitea's metrics or pointing the job at the right path. The LLM alert rules (LlmEndpointDown) are scoped to frank-*, so these probes generate no false pages.

Log aggregation choice

Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:

Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
Loki stores compressed logs with labels; queries run in Grafana's Explore.
No Elasticsearch memory cost, no Kubernetes required.

Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain journald (no centralized query).

15 KiB Raw Blame History