fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES cached tokens (its own HELP line), so the ratio was unbounded — measured 77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never fire on that side. Now cached / (cached + prompt), with the traffic floor counting both terms too; the header documents the unit trap. Also fixed the LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export the llamacpp:* family (verified live), so the alert covers all three endpoints. - shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two namespaces. Every panel reads both (the endpoint picker selects); the KV-pool-fill panel retired 2026-09-16 returned with vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's Llm.* rules; the stale halogen-era description is gone. All 21 panel expressions validated against the live endpoints. - runbook + log + Observability: prefix caching measured, not inferred — reuse happens at 1,648-token block granularity (attention block size padded to the mamba page size); an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules. ./tests/run.sh: 325 pass.
This commit is contained in:
@@ -68,6 +68,27 @@ against the host's 62 GiB and stayed flat through a 4-way burst at 512-token
|
||||
max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the
|
||||
two-replica vision shape, not this one.
|
||||
|
||||
## Prefix caching — measured granularity (2026-09-20)
|
||||
|
||||
`--enable-prefix-caching` is on, and radiance says so at boot: *"Prefix caching
|
||||
in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers
|
||||
is experimental."* The attention block size is **1,648 tokens** (boot log:
|
||||
"Setting attention block size to 1648 tokens …"), padded to the mamba/GDN page
|
||||
size, and reuse happens at that granularity — not per token.
|
||||
|
||||
Measured on the deployed endpoint, not inferred:
|
||||
|
||||
| Probe | Cold | Warm | `vllm:prefix_cache_hits_total` delta |
|
||||
|---|---|---|---|
|
||||
| identical 13,021-token prompt | 2.30 s | 0.68 s (3.4×) | +9,888 of 13,021 queried (76 %) |
|
||||
| identical 2,521-token prompt | 0.51 s | 0.51 s | +0 (under two blocks) |
|
||||
|
||||
Consequence: a shared prefix shorter than about two blocks (≈3.3k tokens)
|
||||
re-prefills in full, so the benefit starts at ~3.3k tokens of shared prefix.
|
||||
The API returns `prompt_tokens_details: null` in this build, so the
|
||||
`vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total` counters
|
||||
(both token-counted) are the only instrument — see [[Observability]].
|
||||
|
||||
## Battery results — why this shape won (2026-09-20)
|
||||
|
||||
`scripts/bench-conversation.py`, 10 real-workload prompts, `max_tokens=512`,
|
||||
|
||||
Reference in New Issue
Block a user