Frank
eb4d9910cb
radiance exports vllm:* metrics (verified on the live endpoint), not llamacpp:*: nothing in llm-fleet could ever see frank-archon. Add a vllm-fleet group - 1 recording + 4 alert (queueing, prefix-cache collapse, preemption, p95 time-to-first-token) keyed to vllm:*. Split the observability seam so llm-fleet rules bind llamacpp:* and vllm-fleet rules bind vllm:*, and neither may evaluate the other's namespace. Drops the retired no-metrics-for-archon doctrine the seam still asserted and the vacuous namespace grep.
202 lines
9.5 KiB
YAML
202 lines
9.5 KiB
YAML
# Alerting and recording rules for the LLM fleet.
|
|
#
|
|
# Two metric namespaces, because two runtime families are scraped:
|
|
#
|
|
# * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen, which
|
|
# re-exports the llama.cpp metric names). Verified from /metrics on
|
|
# 2026-09-16 / 2026-09-20.
|
|
# * vllm:* — frank-archon, the radiance engine on Dankserver since
|
|
# 2026-10-05. It exports vLLM-compatible names, a DIFFERENT namespace, so
|
|
# it is a separate scrape job in prometheus.yml and the rules below are a
|
|
# separate group. (The old claim that its scrape job was deleted is false:
|
|
# it re-joined the day it left the FreeToken engine and is `up` today.)
|
|
# (The netavark published-port quirk that forces the Dankclaw endpoints
|
|
# onto the gateway is same-host only.)
|
|
#
|
|
# Every expression is keyed to a series that EXISTS. Nothing matches on
|
|
# absence: when an endpoint's runtime leaves the fleet, its selectors are
|
|
# deleted rather than left vacuous (the vllm: group went with frank-medium's
|
|
# radiance runtime on 2026-09-26).
|
|
#
|
|
# What llamacpp:* can and cannot see:
|
|
#
|
|
# * KV pressure: llama.cpp exports no fill gauge, so the capacity wall is
|
|
# observed as queueing (llamacpp:requests_deferred).
|
|
# * Prefix reuse: counted in tokens, and prompt_tokens_total EXCLUDES
|
|
# cached tokens (its own HELP line says so), so a reuse fraction divides
|
|
# cached by cached + prompt.
|
|
# * Speculative decoding: drafts and accepted tokens are both counters,
|
|
# so LlmSpecDecodeOff watches the pair.
|
|
# * No latency histograms in llama.cpp, so no p95 alert there; radiance's
|
|
# vllm:* namespace DOES export one, so the vllm-fleet group alerts on it.
|
|
#
|
|
# IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
|
|
# Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
|
|
# alertmanager.container), whose single receiver webhooks the host-side
|
|
# signal-relay user unit, which hands the text to signal-cli under the
|
|
# existing account on this host.
|
|
groups:
|
|
- name: llm-fleet
|
|
interval: 30s
|
|
rules:
|
|
# Recording rules so the dashboard and the collapse alert below share
|
|
# one expression. llamacpp's prompt counter EXCLUDES cached tokens, so
|
|
# the reuse fraction is cached / (cached + prompt).
|
|
- record: job:llm_prefix_cache_reuse:rate5m
|
|
expr: |
|
|
rate(llamacpp:prompt_tokens_cached_total[5m])
|
|
/ clamp_min(rate(llamacpp:prompt_tokens_cached_total[5m])
|
|
+ rate(llamacpp:prompt_tokens_total[5m]), 0.001)
|
|
|
|
# Mean accepted draft length (1.0 = the drafter proposes and nothing is
|
|
# taken, so the draft round is pure overhead).
|
|
- record: job:llm_spec_accept_length:rate5m
|
|
expr: |
|
|
1 + rate(llamacpp:spec_decode_num_accepted_tokens_total[5m])
|
|
/ clamp_min(rate(llamacpp:spec_decode_num_drafts_total[5m]), 0.001)
|
|
|
|
# The capacity wall, in the only form llama.cpp exports: work waiting
|
|
# for a slot. Past here the next large request queues.
|
|
- alert: LlmRequestsQueueing
|
|
expr: llamacpp:requests_deferred > 0
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "{{ $labels.job }}: requests waiting for 10 minutes"
|
|
description: >-
|
|
The endpoint's slot(s) are busy and work is queueing. Either the
|
|
fleet is genuinely busy or one endpoint is carrying it alone. With
|
|
--parallel 1 a second concurrent request always shows up here.
|
|
|
|
# The thrash signature measured on 2026-09-16: requests keep arriving but
|
|
# almost none are served from cache, which is what several concurrent
|
|
# 50-80k-token conversations do to this fleet. One threshold with a
|
|
# traffic floor so an idle endpoint cannot fire it on a 0/0.
|
|
- alert: LlmPrefixCacheCollapse
|
|
expr: >-
|
|
job:llm_prefix_cache_reuse:rate5m < 0.20
|
|
and rate(llamacpp:prompt_tokens_total[5m])
|
|
+ rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05
|
|
for: 15m
|
|
labels:
|
|
severity: info
|
|
annotations:
|
|
summary: "{{ $labels.job }}: prefix reuse below 20 % for 15 minutes"
|
|
description: >-
|
|
Nearly every request is prefilling from scratch. If several large
|
|
contexts are in flight together, this is the capacity wall rather
|
|
than a routing fault.
|
|
|
|
# The speculative path is the one failure no client can see: the server
|
|
# answers normally, just at half the speed, when the drafter is gone.
|
|
# Drafts and accepts are counters, so a rate that stays at zero while
|
|
# the endpoint is serving tokens means nothing is drafted.
|
|
- alert: LlmSpecDecodeOff
|
|
expr: rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
|
|
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "{{ $labels.job }}: serving without speculative decoding"
|
|
description: >-
|
|
Tokens are being produced but no draft has been proposed for 15
|
|
minutes. The llama.cpp load log should carry an "adding
|
|
speculative implementation" line; the endpoint is decoding
|
|
autoregressively.
|
|
|
|
# The endpoint, not a replica: one process serves each port now, so a
|
|
# restart of the unit is a scrape gap, not a failover.
|
|
- alert: LlmEndpointDown
|
|
expr: up{job=~"frank-.*"} == 0
|
|
for: 2m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "{{ $labels.job }}: scrape target down for 2 minutes"
|
|
description: >-
|
|
Prometheus cannot reach the LLM endpoint. frank-zealot and
|
|
frank-dragoon are root units on Dankclaw: `systemctl is-active
|
|
frank-zealot` / `frank-dragoon` (both export llamacpp:* —
|
|
dragoon via halogen's llama.cpp-compatible names). frank-archon
|
|
is a radiance endpoint on Dankserver (`systemctl show
|
|
frank-archon` there) exporting the vllm:* family; the
|
|
`frank-.*` selector covers all three regardless of namespace.
|
|
|
|
# frank-archon (radiance) exports the vLLM-compatible namespace, NOT
|
|
# llamacpp:*. Every rule here is keyed to a series the exporter actually
|
|
# emits (verified from /metrics 2026-10-20: num_requests_waiting,
|
|
# gpu_prefix_cache_hit_rate, num_preemptions_total, prompt_tokens_total,
|
|
# time_to_first_token_seconds). The llamacpp rules above evaluate nothing for
|
|
# this job and these evaluate nothing for llama.cpp/halogen — two clean
|
|
# namespaces, each keyed only to what its runtime exports.
|
|
- name: vllm-fleet
|
|
interval: 30s
|
|
rules:
|
|
# p95 time-to-first-token, shared with the dashboard. When the endpoint
|
|
# is idle every bucket rate is zero, histogram_quantile yields NaN, and
|
|
# neither this nor the alert below can fire on an idle engine.
|
|
- record: job:vllm_ttft_p95:rate5m
|
|
expr: |
|
|
histogram_quantile(0.95, sum by (job, le)
|
|
(rate(vllm:time_to_first_token_seconds_bucket[5m])))
|
|
|
|
# Radiance exports a real queue-depth gauge (llama.cpp only shows the
|
|
# wall as requests_deferred). --max-num-seqs is small, so a sustained
|
|
# wait is either real load or this endpoint carrying the fleet alone.
|
|
- alert: VllmRequestsQueueing
|
|
expr: vllm:num_requests_waiting > 0
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "{{ $labels.job }}: requests waiting for 10 minutes"
|
|
description: >-
|
|
The radiance scheduler has requests queued. Either the fleet is
|
|
genuinely busy or this endpoint is carrying it alone.
|
|
|
|
# Radiance reports prefix reuse directly as a hit-rate gauge — no
|
|
# counter division needed. The token-rate floor keeps an idle engine
|
|
# (hit rate 0 with no traffic) from firing.
|
|
- alert: VllmPrefixCacheCollapse
|
|
expr: >-
|
|
vllm:gpu_prefix_cache_hit_rate < 0.20
|
|
and rate(vllm:prompt_tokens_total[5m]) > 0.05
|
|
for: 15m
|
|
labels:
|
|
severity: info
|
|
annotations:
|
|
summary: "{{ $labels.job }}: prefix hit rate below 20 % for 15 minutes"
|
|
description: >-
|
|
Nearly every request prefilling from scratch. With several large
|
|
contexts in flight this is the capacity wall, not a routing fault.
|
|
|
|
# Preemption is the KV-thrash signature radiance reports that llama.cpp
|
|
# cannot: the engine evicts a running request to admit another. Any
|
|
# sustained rate means the KV cache cannot hold the concurrency in flight.
|
|
- alert: VllmPreempting
|
|
expr: rate(vllm:num_preemptions_total[5m]) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "{{ $labels.job }}: engine preempting requests"
|
|
description: >-
|
|
The engine is evicting running requests to admit new ones — the KV
|
|
cache is undersized for the concurrency in flight.
|
|
|
|
# p95 first-token latency, kept HIGH on purpose: a large-prompt prefill
|
|
# legitimately spends many seconds before token 1, so this fires only on
|
|
# a wedged scheduler or a queue backed up behind one request.
|
|
- alert: VllmTtftSlow
|
|
expr: job:vllm_ttft_p95:rate5m > 120
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "{{ $labels.job }}: p95 time-to-first-token over 120 s"
|
|
description: >-
|
|
First-token latency is pathological, not an honest big prefill.
|
|
Likely a wedged scheduler or one request stalling the queue.
|