# Alerting and recording rules for the LLM fleet. # # Two metric namespaces, because two runtime families are scraped: # # * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen, which # re-exports the llama.cpp metric names). Verified from /metrics on # 2026-09-16 / 2026-09-20. # * vllm:* — frank-archon, the radiance engine on Dankserver since # 2026-10-05. It exports vLLM-compatible names, a DIFFERENT namespace, so # it is a separate scrape job in prometheus.yml and the rules below are a # separate group. (The old claim that its scrape job was deleted is false: # it re-joined the day it left the FreeToken engine and is `up` today.) # (The netavark published-port quirk that forces the Dankclaw endpoints # onto the gateway is same-host only.) # # Every expression is keyed to a series that EXISTS. Nothing matches on # absence: when an endpoint's runtime leaves the fleet, its selectors are # deleted rather than left vacuous (the vllm: group went with frank-medium's # radiance runtime on 2026-09-26). # # What llamacpp:* can and cannot see: # # * KV pressure: llama.cpp exports no fill gauge, so the capacity wall is # observed as queueing (llamacpp:requests_deferred). # * Prefix reuse: counted in tokens, and prompt_tokens_total EXCLUDES # cached tokens (its own HELP line says so), so a reuse fraction divides # cached by cached + prompt. # * Speculative decoding: drafts and accepted tokens are both counters, # so LlmSpecDecodeOff watches the pair. # * No latency histograms in llama.cpp, so no p95 alert there; radiance's # vllm:* namespace DOES export one, so the vllm-fleet group alerts on it. # # IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on # Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/ # alertmanager.container), whose single receiver webhooks the host-side # signal-relay user unit, which hands the text to signal-cli under the # existing account on this host. groups: - name: llm-fleet interval: 30s rules: # Recording rules so the dashboard and the collapse alert below share # one expression. llamacpp's prompt counter EXCLUDES cached tokens, so # the reuse fraction is cached / (cached + prompt). - record: job:llm_prefix_cache_reuse:rate5m expr: | rate(llamacpp:prompt_tokens_cached_total[5m]) / clamp_min(rate(llamacpp:prompt_tokens_cached_total[5m]) + rate(llamacpp:prompt_tokens_total[5m]), 0.001) # Mean accepted draft length (1.0 = the drafter proposes and nothing is # taken, so the draft round is pure overhead). - record: job:llm_spec_accept_length:rate5m expr: | 1 + rate(llamacpp:spec_decode_num_accepted_tokens_total[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total[5m]), 0.001) # The capacity wall, in the only form llama.cpp exports: work waiting # for a slot. Past here the next large request queues. - alert: LlmRequestsQueueing expr: llamacpp:requests_deferred > 0 for: 10m labels: severity: warning annotations: summary: "{{ $labels.job }}: requests waiting for 10 minutes" description: >- The endpoint's slot(s) are busy and work is queueing. Either the fleet is genuinely busy or one endpoint is carrying it alone. With --parallel 1 a second concurrent request always shows up here. # The thrash signature measured on 2026-09-16: requests keep arriving but # almost none are served from cache, which is what several concurrent # 50-80k-token conversations do to this fleet. One threshold with a # traffic floor so an idle endpoint cannot fire it on a 0/0. - alert: LlmPrefixCacheCollapse expr: >- job:llm_prefix_cache_reuse:rate5m < 0.20 and rate(llamacpp:prompt_tokens_total[5m]) + rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05 for: 15m labels: severity: info annotations: summary: "{{ $labels.job }}: prefix reuse below 20 % for 15 minutes" description: >- Nearly every request is prefilling from scratch. If several large contexts are in flight together, this is the capacity wall rather than a routing fault. # The speculative path is the one failure no client can see: the server # answers normally, just at half the speed, when the drafter is gone. # Drafts and accepts are counters, so a rate that stays at zero while # the endpoint is serving tokens means nothing is drafted. - alert: LlmSpecDecodeOff expr: rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0 and rate(llamacpp:tokens_predicted_total[5m]) > 0.1 for: 15m labels: severity: warning annotations: summary: "{{ $labels.job }}: serving without speculative decoding" description: >- Tokens are being produced but no draft has been proposed for 15 minutes. The llama.cpp load log should carry an "adding speculative implementation" line; the endpoint is decoding autoregressively. # The endpoint, not a replica: one process serves each port now, so a # restart of the unit is a scrape gap, not a failover. - alert: LlmEndpointDown expr: up{job=~"frank-.*"} == 0 for: 2m labels: severity: critical annotations: summary: "{{ $labels.job }}: scrape target down for 2 minutes" description: >- Prometheus cannot reach the LLM endpoint. frank-zealot and frank-dragoon are root units on Dankclaw: `systemctl is-active frank-zealot` / `frank-dragoon` (both export llamacpp:* — dragoon via halogen's llama.cpp-compatible names). frank-archon is a radiance endpoint on Dankserver (`systemctl show frank-archon` there) exporting the vllm:* family; the `frank-.*` selector covers all three regardless of namespace. # frank-archon (radiance) exports the vLLM-compatible namespace, NOT # llamacpp:*. Every rule here is keyed to a series the exporter actually # emits (verified from /metrics 2026-10-20: num_requests_waiting, # gpu_prefix_cache_hit_rate, num_preemptions_total, prompt_tokens_total, # time_to_first_token_seconds). The llamacpp rules above evaluate nothing for # this job and these evaluate nothing for llama.cpp/halogen — two clean # namespaces, each keyed only to what its runtime exports. - name: vllm-fleet interval: 30s rules: # p95 time-to-first-token, shared with the dashboard. When the endpoint # is idle every bucket rate is zero, histogram_quantile yields NaN, and # neither this nor the alert below can fire on an idle engine. - record: job:vllm_ttft_p95:rate5m expr: | histogram_quantile(0.95, sum by (job, le) (rate(vllm:time_to_first_token_seconds_bucket[5m]))) # Radiance exports a real queue-depth gauge (llama.cpp only shows the # wall as requests_deferred). --max-num-seqs is small, so a sustained # wait is either real load or this endpoint carrying the fleet alone. - alert: VllmRequestsQueueing expr: vllm:num_requests_waiting > 0 for: 10m labels: severity: warning annotations: summary: "{{ $labels.job }}: requests waiting for 10 minutes" description: >- The radiance scheduler has requests queued. Either the fleet is genuinely busy or this endpoint is carrying it alone. # Radiance reports prefix reuse directly as a hit-rate gauge — no # counter division needed. The token-rate floor keeps an idle engine # (hit rate 0 with no traffic) from firing. - alert: VllmPrefixCacheCollapse expr: >- vllm:gpu_prefix_cache_hit_rate < 0.20 and rate(vllm:prompt_tokens_total[5m]) > 0.05 for: 15m labels: severity: info annotations: summary: "{{ $labels.job }}: prefix hit rate below 20 % for 15 minutes" description: >- Nearly every request prefilling from scratch. With several large contexts in flight this is the capacity wall, not a routing fault. # Preemption is the KV-thrash signature radiance reports that llama.cpp # cannot: the engine evicts a running request to admit another. Any # sustained rate means the KV cache cannot hold the concurrency in flight. - alert: VllmPreempting expr: rate(vllm:num_preemptions_total[5m]) > 0 for: 5m labels: severity: warning annotations: summary: "{{ $labels.job }}: engine preempting requests" description: >- The engine is evicting running requests to admit new ones — the KV cache is undersized for the concurrency in flight. # p95 first-token latency, kept HIGH on purpose: a large-prompt prefill # legitimately spends many seconds before token 1, so this fires only on # a wedged scheduler or a queue backed up behind one request. - alert: VllmTtftSlow expr: job:vllm_ttft_p95:rate5m > 120 for: 10m labels: severity: warning annotations: summary: "{{ $labels.job }}: p95 time-to-first-token over 120 s" description: >- First-token latency is pathological, not an honest big prefill. Likely a wedged scheduler or one request stalling the queue.