feat(observability): add vllm-fleet rule group for frank-archon
radiance exports vllm:* metrics (verified on the live endpoint), not llamacpp:*: nothing in llm-fleet could ever see frank-archon. Add a vllm-fleet group - 1 recording + 4 alert (queueing, prefix-cache collapse, preemption, p95 time-to-first-token) keyed to vllm:*. Split the observability seam so llm-fleet rules bind llamacpp:* and vllm-fleet rules bind vllm:*, and neither may evaluate the other's namespace. Drops the retired no-metrics-for-archon doctrine the seam still asserted and the vacuous namespace grep.
This commit is contained in:
Frank
@@ -1,10 +1,15 @@
|
||||
# Alerting and recording rules for the LLM fleet.
|
||||
#
|
||||
# One metric namespace: llamacpp:* — frank-zealot and frank-dragoon, both
|
||||
# llama.cpp with /metrics (the family verified from /metrics on 2026-09-16
|
||||
# and 2026-09-20). frank-archon LEFT the family 2026-10-05: FreeToken (the
|
||||
# PR587 ROCm engine) has no /metrics endpoint, so its scrape job was
|
||||
# deleted rather than left never-green.
|
||||
# Two metric namespaces, because two runtime families are scraped:
|
||||
#
|
||||
# * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen, which
|
||||
# re-exports the llama.cpp metric names). Verified from /metrics on
|
||||
# 2026-09-16 / 2026-09-20.
|
||||
# * vllm:* — frank-archon, the radiance engine on Dankserver since
|
||||
# 2026-10-05. It exports vLLM-compatible names, a DIFFERENT namespace, so
|
||||
# it is a separate scrape job in prometheus.yml and the rules below are a
|
||||
# separate group. (The old claim that its scrape job was deleted is false:
|
||||
# it re-joined the day it left the FreeToken engine and is `up` today.)
|
||||
# (The netavark published-port quirk that forces the Dankclaw endpoints
|
||||
# onto the gateway is same-host only.)
|
||||
#
|
||||
@@ -22,7 +27,8 @@
|
||||
# cached by cached + prompt.
|
||||
# * Speculative decoding: drafts and accepted tokens are both counters,
|
||||
# so LlmSpecDecodeOff watches the pair.
|
||||
# * No latency histograms exist, so there is no p95 alert.
|
||||
# * No latency histograms in llama.cpp, so no p95 alert there; radiance's
|
||||
# vllm:* namespace DOES export one, so the vllm-fleet group alerts on it.
|
||||
#
|
||||
# IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
|
||||
# Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
|
||||
@@ -112,6 +118,84 @@ groups:
|
||||
description: >-
|
||||
Prometheus cannot reach the LLM endpoint. frank-zealot and
|
||||
frank-dragoon are root units on Dankclaw: `systemctl is-active
|
||||
frank-zealot` / `frank-dragoon`. Both scraped endpoints export
|
||||
the llamacpp:* family. (frank-archon left the scrape set
|
||||
2026-10-05 — FreeToken exports no /metrics.)
|
||||
frank-zealot` / `frank-dragoon` (both export llamacpp:* —
|
||||
dragoon via halogen's llama.cpp-compatible names). frank-archon
|
||||
is a radiance endpoint on Dankserver (`systemctl show
|
||||
frank-archon` there) exporting the vllm:* family; the
|
||||
`frank-.*` selector covers all three regardless of namespace.
|
||||
|
||||
# frank-archon (radiance) exports the vLLM-compatible namespace, NOT
|
||||
# llamacpp:*. Every rule here is keyed to a series the exporter actually
|
||||
# emits (verified from /metrics 2026-10-20: num_requests_waiting,
|
||||
# gpu_prefix_cache_hit_rate, num_preemptions_total, prompt_tokens_total,
|
||||
# time_to_first_token_seconds). The llamacpp rules above evaluate nothing for
|
||||
# this job and these evaluate nothing for llama.cpp/halogen — two clean
|
||||
# namespaces, each keyed only to what its runtime exports.
|
||||
- name: vllm-fleet
|
||||
interval: 30s
|
||||
rules:
|
||||
# p95 time-to-first-token, shared with the dashboard. When the endpoint
|
||||
# is idle every bucket rate is zero, histogram_quantile yields NaN, and
|
||||
# neither this nor the alert below can fire on an idle engine.
|
||||
- record: job:vllm_ttft_p95:rate5m
|
||||
expr: |
|
||||
histogram_quantile(0.95, sum by (job, le)
|
||||
(rate(vllm:time_to_first_token_seconds_bucket[5m])))
|
||||
|
||||
# Radiance exports a real queue-depth gauge (llama.cpp only shows the
|
||||
# wall as requests_deferred). --max-num-seqs is small, so a sustained
|
||||
# wait is either real load or this endpoint carrying the fleet alone.
|
||||
- alert: VllmRequestsQueueing
|
||||
expr: vllm:num_requests_waiting > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: requests waiting for 10 minutes"
|
||||
description: >-
|
||||
The radiance scheduler has requests queued. Either the fleet is
|
||||
genuinely busy or this endpoint is carrying it alone.
|
||||
|
||||
# Radiance reports prefix reuse directly as a hit-rate gauge — no
|
||||
# counter division needed. The token-rate floor keeps an idle engine
|
||||
# (hit rate 0 with no traffic) from firing.
|
||||
- alert: VllmPrefixCacheCollapse
|
||||
expr: >-
|
||||
vllm:gpu_prefix_cache_hit_rate < 0.20
|
||||
and rate(vllm:prompt_tokens_total[5m]) > 0.05
|
||||
for: 15m
|
||||
labels:
|
||||
severity: info
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: prefix hit rate below 20 % for 15 minutes"
|
||||
description: >-
|
||||
Nearly every request prefilling from scratch. With several large
|
||||
contexts in flight this is the capacity wall, not a routing fault.
|
||||
|
||||
# Preemption is the KV-thrash signature radiance reports that llama.cpp
|
||||
# cannot: the engine evicts a running request to admit another. Any
|
||||
# sustained rate means the KV cache cannot hold the concurrency in flight.
|
||||
- alert: VllmPreempting
|
||||
expr: rate(vllm:num_preemptions_total[5m]) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: engine preempting requests"
|
||||
description: >-
|
||||
The engine is evicting running requests to admit new ones — the KV
|
||||
cache is undersized for the concurrency in flight.
|
||||
|
||||
# p95 first-token latency, kept HIGH on purpose: a large-prompt prefill
|
||||
# legitimately spends many seconds before token 1, so this fires only on
|
||||
# a wedged scheduler or a queue backed up behind one request.
|
||||
- alert: VllmTtftSlow
|
||||
expr: job:vllm_ttft_p95:rate5m > 120
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: p95 time-to-first-token over 120 s"
|
||||
description: >-
|
||||
First-token latency is pathological, not an honest big prefill.
|
||||
Likely a wedged scheduler or one request stalling the queue.
|
||||
|
||||
Reference in New Issue
Block a user