fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES cached tokens (its own HELP line), so the ratio was unbounded — measured 77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never fire on that side. Now cached / (cached + prompt), with the traffic floor counting both terms too; the header documents the unit trap. Also fixed the LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export the llamacpp:* family (verified live), so the alert covers all three endpoints. - shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two namespaces. Every panel reads both (the endpoint picker selects); the KV-pool-fill panel retired 2026-09-16 returned with vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's Llm.* rules; the stale halogen-era description is gone. All 21 panel expressions validated against the live endpoints. - runbook + log + Observability: prefix caching measured, not inferred — reuse happens at 1,648-token block granularity (attention block size padded to the mamba page size); an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules. ./tests/run.sh: 325 pass.
This commit is contained in:
@@ -16,8 +16,11 @@
|
||||
# * KV pressure. vLLM exports a fill gauge (vllm:kv_cache_usage_perc) and a
|
||||
# preemption counter; llama.cpp exports neither, so its capacity wall is
|
||||
# observed as queueing (llamacpp:requests_deferred).
|
||||
# * Prefix reuse. Both export a hits/queries pair, in tokens; the two
|
||||
# recording rules below keep the units straight and the collapse alert
|
||||
# * Prefix reuse. Both export a hits/queries pair counted in tokens, but the
|
||||
# units are NOT interchangeable: llama.cpp's prompt_tokens_total EXCLUDES
|
||||
# cached tokens (its own HELP line says so), so that side must divide by
|
||||
# cached + prompt to get a fraction, while vLLM divides hits by queries.
|
||||
# The two recording rules below keep that straight, and the collapse alert
|
||||
# watches both sides with one threshold.
|
||||
# * Speculative decoding. Both export drafts and accepted tokens, so
|
||||
# LlmSpecDecodeOff covers both runtimes.
|
||||
@@ -42,7 +45,8 @@ groups:
|
||||
- record: job:llm_prefix_cache_reuse:rate5m
|
||||
expr: |
|
||||
rate(llamacpp:prompt_tokens_cached_total[5m])
|
||||
/ clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001)
|
||||
/ clamp_min(rate(llamacpp:prompt_tokens_cached_total[5m])
|
||||
+ rate(llamacpp:prompt_tokens_total[5m]), 0.001)
|
||||
|
||||
- record: job:llm_vllm_prefix_cache_reuse:rate5m
|
||||
expr: |
|
||||
@@ -112,7 +116,8 @@ groups:
|
||||
- alert: LlmPrefixCacheCollapse
|
||||
expr: >-
|
||||
(job:llm_prefix_cache_reuse:rate5m < 0.20
|
||||
and rate(llamacpp:prompt_tokens_total[5m]) > 0.05)
|
||||
and rate(llamacpp:prompt_tokens_total[5m])
|
||||
+ rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05)
|
||||
or (job:llm_vllm_prefix_cache_reuse:rate5m < 0.20
|
||||
and rate(vllm:prefix_cache_queries_total[5m]) > 1)
|
||||
for: 15m
|
||||
@@ -162,6 +167,6 @@ groups:
|
||||
rootless quadlet: `systemctl --user is-active frank-medium` on
|
||||
Dankserver, then `podman ps -a --filter name=frank-medium`.
|
||||
frank-small is a root unit: `systemctl is-active frank-small` on
|
||||
Dankclaw. frank-large is not scraped at all — its CIRU runtime
|
||||
serves no Prometheus text endpoint — so this alert can only ever
|
||||
name the endpoints that do export one.
|
||||
Dankclaw. All three endpoints are scraped: frank-large's CIRU
|
||||
llama.cpp build does export the llamacpp:* family (verified
|
||||
2026-09-20), so this alert covers the whole fleet.
|
||||
|
||||
Reference in New Issue
Block a user