fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity

- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
  divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
  cached tokens (its own HELP line), so the ratio was unbounded — measured
  77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
  fire on that side. Now cached / (cached + prompt), with the traffic floor
  counting both terms too; the header documents the unit trap. Also fixed the
  LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
  the llamacpp:* family (verified live), so the alert covers all three
  endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
  namespaces. Every panel reads both (the endpoint picker selects); the
  KV-pool-fill panel retired 2026-09-16 returned with
  vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
  histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
  Llm.* rules; the stale halogen-era description is gone. All 21 panel
  expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
  reuse happens at 1,648-token block granularity (attention block size padded
  to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
  to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
  registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.

./tests/run.sh: 325 pass.
This commit is contained in:
Frank
2026-09-20 19:53:14 -07:00
parent 08f1c7ad09
commit c42780f9f3
5 changed files with 220 additions and 137 deletions
+12 -7
View File
@@ -16,8 +16,11 @@
# * KV pressure. vLLM exports a fill gauge (vllm:kv_cache_usage_perc) and a
# preemption counter; llama.cpp exports neither, so its capacity wall is
# observed as queueing (llamacpp:requests_deferred).
# * Prefix reuse. Both export a hits/queries pair, in tokens; the two
# recording rules below keep the units straight and the collapse alert
# * Prefix reuse. Both export a hits/queries pair counted in tokens, but the
# units are NOT interchangeable: llama.cpp's prompt_tokens_total EXCLUDES
# cached tokens (its own HELP line says so), so that side must divide by
# cached + prompt to get a fraction, while vLLM divides hits by queries.
# The two recording rules below keep that straight, and the collapse alert
# watches both sides with one threshold.
# * Speculative decoding. Both export drafts and accepted tokens, so
# LlmSpecDecodeOff covers both runtimes.
@@ -42,7 +45,8 @@ groups:
- record: job:llm_prefix_cache_reuse:rate5m
expr: |
rate(llamacpp:prompt_tokens_cached_total[5m])
/ clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001)
/ clamp_min(rate(llamacpp:prompt_tokens_cached_total[5m])
+ rate(llamacpp:prompt_tokens_total[5m]), 0.001)
- record: job:llm_vllm_prefix_cache_reuse:rate5m
expr: |
@@ -112,7 +116,8 @@ groups:
- alert: LlmPrefixCacheCollapse
expr: >-
(job:llm_prefix_cache_reuse:rate5m < 0.20
and rate(llamacpp:prompt_tokens_total[5m]) > 0.05)
and rate(llamacpp:prompt_tokens_total[5m])
+ rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05)
or (job:llm_vllm_prefix_cache_reuse:rate5m < 0.20
and rate(vllm:prefix_cache_queries_total[5m]) > 1)
for: 15m
@@ -162,6 +167,6 @@ groups:
rootless quadlet: `systemctl --user is-active frank-medium` on
Dankserver, then `podman ps -a --filter name=frank-medium`.
frank-small is a root unit: `systemctl is-active frank-small` on
Dankclaw. frank-large is not scraped at all — its CIRU runtime
serves no Prometheus text endpoint — so this alert can only ever
name the endpoints that do export one.
Dankclaw. All three endpoints are scraped: frank-large's CIRU
llama.cpp build does export the llamacpp:* family (verified
2026-09-20), so this alert covers the whole fleet.