fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity

- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
  divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
  cached tokens (its own HELP line), so the ratio was unbounded — measured
  77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
  fire on that side. Now cached / (cached + prompt), with the traffic floor
  counting both terms too; the header documents the unit trap. Also fixed the
  LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
  the llamacpp:* family (verified live), so the alert covers all three
  endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
  namespaces. Every panel reads both (the endpoint picker selects); the
  KV-pool-fill panel retired 2026-09-16 returned with
  vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
  histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
  Llm.* rules; the stale halogen-era description is gone. All 21 panel
  expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
  reuse happens at 1,648-token block granularity (attention block size padded
  to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
  to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
  registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.

./tests/run.sh: 325 pass.
This commit is contained in:
Frank
2026-09-20 19:53:14 -07:00
parent 08f1c7ad09
commit c42780f9f3
5 changed files with 220 additions and 137 deletions
+15 -5
View File
@@ -46,14 +46,24 @@ timestamp: 2026-09-14
≤ 262,144).
- **Monitoring cut over with the runtime.** `shared/prometheus/rules/llm-fleet.yml`
now spans two namespaces — `vllm:*` for frank-medium, `llamacpp:*` for the
Dankclaw pair — with 7 alerts + 2 recording rules; KV-saturation and preemption
Dankclaw pair — with 6 alerts + 3 recording rules; KV-saturation and preemption
pressure returned (vLLM exports both), and the two engine-independent rules OR
the families with per-family traffic floors. Every expression validated against
live Prometheus. **Two scrape targets were red by construction and are fixed**:
the families with per-family traffic floors. One semantics trap fixed the same
day: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP
line), so the reuse rule divides by `cached + prompt` — the naive
`cached/prompt` reads 77 on a fully-warm endpoint and can never show a
collapse. **Two scrape targets were red by construction and are fixed**:
frank-small and frank-large are *host processes* on Dankclaw and netavark DNATs
only published ports, so both are now addressed as `host.containers.internal`.
Known gap: the provisioned Grafana dashboard is still keyed to `llamacpp:*`, so
frank-medium's panels are blank pending a re-cut.
The provisioned Grafana dashboard was re-cut the same day: every panel reads
both namespaces (the endpoint picker selects), the KV-fill panel returned with
`vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM
histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every
rule and dashboard expression validated against the live endpoints. **Prefix
caching measured, not inferred**: reuse happens at 1,648-token block granularity
(the attention block size is padded to the mamba page size) — an identical
13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %)
reused, while a 2,521-token prompt registered zero hits (under two blocks).
- **Network decision.** `eno2` (5GbE, 192.168.0.247) kept, `eno1` pinned primary
via metric 100 vs 200 in `scripts/dankserver-route-primary.sh`; applied with
`nmcli device reapply` (there is no `nmcli connection reapply` in NM 1.54).