fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES cached tokens (its own HELP line), so the ratio was unbounded — measured 77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never fire on that side. Now cached / (cached + prompt), with the traffic floor counting both terms too; the header documents the unit trap. Also fixed the LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export the llamacpp:* family (verified live), so the alert covers all three endpoints. - shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two namespaces. Every panel reads both (the endpoint picker selects); the KV-pool-fill panel retired 2026-09-16 returned with vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's Llm.* rules; the stale halogen-era description is gone. All 21 panel expressions validated against the live endpoints. - runbook + log + Observability: prefix caching measured, not inferred — reuse happens at 1,648-token block granularity (attention block size padded to the mamba page size); an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules. ./tests/run.sh: 325 pass.
This commit is contained in:
@@ -46,14 +46,24 @@ timestamp: 2026-09-14
|
||||
≤ 262,144).
|
||||
- **Monitoring cut over with the runtime.** `shared/prometheus/rules/llm-fleet.yml`
|
||||
now spans two namespaces — `vllm:*` for frank-medium, `llamacpp:*` for the
|
||||
Dankclaw pair — with 7 alerts + 2 recording rules; KV-saturation and preemption
|
||||
Dankclaw pair — with 6 alerts + 3 recording rules; KV-saturation and preemption
|
||||
pressure returned (vLLM exports both), and the two engine-independent rules OR
|
||||
the families with per-family traffic floors. Every expression validated against
|
||||
live Prometheus. **Two scrape targets were red by construction and are fixed**:
|
||||
the families with per-family traffic floors. One semantics trap fixed the same
|
||||
day: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP
|
||||
line), so the reuse rule divides by `cached + prompt` — the naive
|
||||
`cached/prompt` reads 77 on a fully-warm endpoint and can never show a
|
||||
collapse. **Two scrape targets were red by construction and are fixed**:
|
||||
frank-small and frank-large are *host processes* on Dankclaw and netavark DNATs
|
||||
only published ports, so both are now addressed as `host.containers.internal`.
|
||||
Known gap: the provisioned Grafana dashboard is still keyed to `llamacpp:*`, so
|
||||
frank-medium's panels are blank pending a re-cut.
|
||||
The provisioned Grafana dashboard was re-cut the same day: every panel reads
|
||||
both namespaces (the endpoint picker selects), the KV-fill panel returned with
|
||||
`vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM
|
||||
histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every
|
||||
rule and dashboard expression validated against the live endpoints. **Prefix
|
||||
caching measured, not inferred**: reuse happens at 1,648-token block granularity
|
||||
(the attention block size is padded to the mamba page size) — an identical
|
||||
13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %)
|
||||
reused, while a 2,521-token prompt registered zero hits (under two blocks).
|
||||
- **Network decision.** `eno2` (5GbE, 192.168.0.247) kept, `eno1` pinned primary
|
||||
via metric 100 vs 200 in `scripts/dankserver-route-primary.sh`; applied with
|
||||
`nmcli device reapply` (there is no `nmcli connection reapply` in NM 1.54).
|
||||
|
||||
Reference in New Issue
Block a user