fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity

- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
  divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
  cached tokens (its own HELP line), so the ratio was unbounded — measured
  77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
  fire on that side. Now cached / (cached + prompt), with the traffic floor
  counting both terms too; the header documents the unit trap. Also fixed the
  LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
  the llamacpp:* family (verified live), so the alert covers all three
  endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
  namespaces. Every panel reads both (the endpoint picker selects); the
  KV-pool-fill panel retired 2026-09-16 returned with
  vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
  histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
  Llm.* rules; the stale halogen-era description is gone. All 21 panel
  expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
  reuse happens at 1,648-token block granularity (attention block size padded
  to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
  to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
  registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.

./tests/run.sh: 325 pass.

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -41,12 +41,12 @@ Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in
 - **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
 - **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
 - **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the relay binds the host's LAN IP (192.168.0.143:18093, the one address netavark delivers that dial to), with an iptables set admitting loopback, the host and the podman bridge subnet only. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
-- **2026-09-20: the rule file now spans two metric namespaces.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — seven alerts and two recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. **The provisioned dashboard is the known gap:** its panels are still keyed to `llamacpp:*` series, so frank-medium's row went blank at the cutover while frank-small's and frank-large's still draw. Re-cut pending — the KV-fill and latency panels that were retired for llama.cpp have vLLM equivalents again (`vllm:kv_cache_usage_perc`, `vllm:inter_token_latency_seconds`).
+- **2026-09-20: the rule file now spans two metric namespaces, and the dashboard was re-cut the same day.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — six alerts and three recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. One semantics trap, measured 2026-09-20: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP line), so the llama.cpp reuse rule divides by `cached + prompt` — the naive `cached/prompt` ratio reads 77 on a fully-warm endpoint and can never show a collapse. The provisioned dashboard's panels now read both namespaces (the endpoint picker selects between them); the KV-pool-fill panel retired 2026-09-16 returned with `vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every expression in the rules and the dashboard was validated against the live endpoints on 2026-09-20.
 - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
 - **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
 - **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
 
-The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, five rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent.
+The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, nine rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent.
 
 ## Deployment
 

serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md

@@ -68,6 +68,27 @@ against the host's 62 GiB and stayed flat through a 4-way burst at 512-token
 max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the
 two-replica vision shape, not this one.
 
+## Prefix caching — measured granularity (2026-09-20)
+
+`--enable-prefix-caching` is on, and radiance says so at boot: *"Prefix caching
+in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers
+is experimental."* The attention block size is **1,648 tokens** (boot log:
+"Setting attention block size to 1648 tokens …"), padded to the mamba/GDN page
+size, and reuse happens at that granularity — not per token.
+
+Measured on the deployed endpoint, not inferred:
+
+| Probe | Cold | Warm | `vllm:prefix_cache_hits_total` delta |
+|---|---|---|---|
+| identical 13,021-token prompt | 2.30 s | 0.68 s (3.4×) | +9,888 of 13,021 queried (76 %) |
+| identical 2,521-token prompt | 0.51 s | 0.51 s | +0 (under two blocks) |
+
+Consequence: a shared prefix shorter than about two blocks (≈3.3k tokens)
+re-prefills in full, so the benefit starts at ~3.3k tokens of shared prefix.
+The API returns `prompt_tokens_details: null` in this build, so the
+`vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total` counters
+(both token-counted) are the only instrument — see [[Observability]].
+
 ## Battery results — why this shape won (2026-09-20)
 
 `scripts/bench-conversation.py`, 10 real-workload prompts, `max_tokens=512`,

serverAppConfigs-docs/log.md

@@ -46,14 +46,24 @@ timestamp: 2026-09-14
   ≤ 262,144).
 - **Monitoring cut over with the runtime.** `shared/prometheus/rules/llm-fleet.yml`
   now spans two namespaces — `vllm:*` for frank-medium, `llamacpp:*` for the
-  Dankclaw pair — with 7 alerts + 2 recording rules; KV-saturation and preemption
+  Dankclaw pair — with 6 alerts + 3 recording rules; KV-saturation and preemption
   pressure returned (vLLM exports both), and the two engine-independent rules OR
-  the families with per-family traffic floors. Every expression validated against
-  live Prometheus. **Two scrape targets were red by construction and are fixed**:
+  the families with per-family traffic floors. One semantics trap fixed the same
+  day: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP
+  line), so the reuse rule divides by `cached + prompt` — the naive
+  `cached/prompt` reads 77 on a fully-warm endpoint and can never show a
+  collapse. **Two scrape targets were red by construction and are fixed**:
   frank-small and frank-large are *host processes* on Dankclaw and netavark DNATs
   only published ports, so both are now addressed as `host.containers.internal`.
-  Known gap: the provisioned Grafana dashboard is still keyed to `llamacpp:*`, so
-  frank-medium's panels are blank pending a re-cut.
+  The provisioned Grafana dashboard was re-cut the same day: every panel reads
+  both namespaces (the endpoint picker selects), the KV-fill panel returned with
+  `vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM
+  histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every
+  rule and dashboard expression validated against the live endpoints. **Prefix
+  caching measured, not inferred**: reuse happens at 1,648-token block granularity
+  (the attention block size is padded to the mamba page size) — an identical
+  13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %)
+  reused, while a 2,521-token prompt registered zero hits (under two blocks).
 - **Network decision.** `eno2` (5GbE, 192.168.0.247) kept, `eno1` pinned primary
   via metric 100 vs 200 in `scripts/dankserver-route-primary.sh`; applied with
   `nmcli device reapply` (there is no `nmcli connection reapply` in NM 1.54).