fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity

- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
  divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
  cached tokens (its own HELP line), so the ratio was unbounded — measured
  77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
  fire on that side. Now cached / (cached + prompt), with the traffic floor
  counting both terms too; the header documents the unit trap. Also fixed the
  LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
  the llamacpp:* family (verified live), so the alert covers all three
  endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
  namespaces. Every panel reads both (the endpoint picker selects); the
  KV-pool-fill panel retired 2026-09-16 returned with
  vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
  histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
  Llm.* rules; the stale halogen-era description is gone. All 21 panel
  expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
  reuse happens at 1,648-token block granularity (attention block size padded
  to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
  to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
  registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.

./tests/run.sh: 325 pass.
This commit is contained in:
Frank
2026-09-20 19:53:14 -07:00
parent 08f1c7ad09
commit c42780f9f3
5 changed files with 220 additions and 137 deletions
@@ -68,6 +68,27 @@ against the host's 62 GiB and stayed flat through a 4-way burst at 512-token
max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the
two-replica vision shape, not this one.
## Prefix caching — measured granularity (2026-09-20)
`--enable-prefix-caching` is on, and radiance says so at boot: *"Prefix caching
in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers
is experimental."* The attention block size is **1,648 tokens** (boot log:
"Setting attention block size to 1648 tokens …"), padded to the mamba/GDN page
size, and reuse happens at that granularity — not per token.
Measured on the deployed endpoint, not inferred:
| Probe | Cold | Warm | `vllm:prefix_cache_hits_total` delta |
|---|---|---|---|
| identical 13,021-token prompt | 2.30 s | 0.68 s (3.4×) | +9,888 of 13,021 queried (76 %) |
| identical 2,521-token prompt | 0.51 s | 0.51 s | +0 (under two blocks) |
Consequence: a shared prefix shorter than about two blocks (≈3.3k tokens)
re-prefills in full, so the benefit starts at ~3.3k tokens of shared prefix.
The API returns `prompt_tokens_details: null` in this build, so the
`vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total` counters
(both token-counted) are the only instrument — see [[Observability]].
## Battery results — why this shape won (2026-09-20)
`scripts/bench-conversation.py`, 10 real-workload prompts, `max_tokens=512`,