diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 6bc076a..d67be88 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -41,12 +41,12 @@ Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in - **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage … - **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus. - **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the relay binds the host's LAN IP (192.168.0.143:18093, the one address netavark delivers that dial to), with an iptables set admitting loopback, the host and the podman bridge subnet only. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces. -- **2026-09-20: the rule file now spans two metric namespaces.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — seven alerts and two recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. **The provisioned dashboard is the known gap:** its panels are still keyed to `llamacpp:*` series, so frank-medium's row went blank at the cutover while frank-small's and frank-large's still draw. Re-cut pending — the KV-fill and latency panels that were retired for llama.cpp have vLLM equivalents again (`vllm:kv_cache_usage_perc`, `vllm:inter_token_latency_seconds`). +- **2026-09-20: the rule file now spans two metric namespaces, and the dashboard was re-cut the same day.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — six alerts and three recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. One semantics trap, measured 2026-09-20: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP line), so the llama.cpp reuse rule divides by `cached + prompt` — the naive `cached/prompt` ratio reads 77 on a fully-warm endpoint and can never show a collapse. The provisioned dashboard's panels now read both namespaces (the endpoint picker selects between them); the KV-pool-fill panel retired 2026-09-16 returned with `vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every expression in the rules and the dashboard was validated against the live endpoints on 2026-09-20. - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it. - **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics. - **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live. -The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, five rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent. +The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, nine rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent. ## Deployment diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 192451e..729631a 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -68,6 +68,27 @@ against the host's 62 GiB and stayed flat through a 4-way burst at 512-token max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the two-replica vision shape, not this one. +## Prefix caching — measured granularity (2026-09-20) + +`--enable-prefix-caching` is on, and radiance says so at boot: *"Prefix caching +in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers +is experimental."* The attention block size is **1,648 tokens** (boot log: +"Setting attention block size to 1648 tokens …"), padded to the mamba/GDN page +size, and reuse happens at that granularity — not per token. + +Measured on the deployed endpoint, not inferred: + +| Probe | Cold | Warm | `vllm:prefix_cache_hits_total` delta | +|---|---|---|---| +| identical 13,021-token prompt | 2.30 s | 0.68 s (3.4×) | +9,888 of 13,021 queried (76 %) | +| identical 2,521-token prompt | 0.51 s | 0.51 s | +0 (under two blocks) | + +Consequence: a shared prefix shorter than about two blocks (≈3.3k tokens) +re-prefills in full, so the benefit starts at ~3.3k tokens of shared prefix. +The API returns `prompt_tokens_details: null` in this build, so the +`vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total` counters +(both token-counted) are the only instrument — see [[Observability]]. + ## Battery results — why this shape won (2026-09-20) `scripts/bench-conversation.py`, 10 real-workload prompts, `max_tokens=512`, diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index d24af48..f4ceac0 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -46,14 +46,24 @@ timestamp: 2026-09-14 ≤ 262,144). - **Monitoring cut over with the runtime.** `shared/prometheus/rules/llm-fleet.yml` now spans two namespaces — `vllm:*` for frank-medium, `llamacpp:*` for the - Dankclaw pair — with 7 alerts + 2 recording rules; KV-saturation and preemption + Dankclaw pair — with 6 alerts + 3 recording rules; KV-saturation and preemption pressure returned (vLLM exports both), and the two engine-independent rules OR - the families with per-family traffic floors. Every expression validated against - live Prometheus. **Two scrape targets were red by construction and are fixed**: + the families with per-family traffic floors. One semantics trap fixed the same + day: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP + line), so the reuse rule divides by `cached + prompt` — the naive + `cached/prompt` reads 77 on a fully-warm endpoint and can never show a + collapse. **Two scrape targets were red by construction and are fixed**: frank-small and frank-large are *host processes* on Dankclaw and netavark DNATs only published ports, so both are now addressed as `host.containers.internal`. - Known gap: the provisioned Grafana dashboard is still keyed to `llamacpp:*`, so - frank-medium's panels are blank pending a re-cut. + The provisioned Grafana dashboard was re-cut the same day: every panel reads + both namespaces (the endpoint picker selects), the KV-fill panel returned with + `vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM + histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every + rule and dashboard expression validated against the live endpoints. **Prefix + caching measured, not inferred**: reuse happens at 1,648-token block granularity + (the attention block size is padded to the mamba page size) — an identical + 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) + reused, while a 2,521-token prompt registered zero hits (under two blocks). - **Network decision.** `eno2` (5GbE, 192.168.0.247) kept, `eno1` pinned primary via metric 100 vs 200 in `scripts/dankserver-route-primary.sh`; applied with `nmcli device reapply` (there is no `nmcli connection reapply` in NM 1.54). diff --git a/shared/grafana/provisioning/dashboards/llm-fleet.json b/shared/grafana/provisioning/dashboards/llm-fleet.json index f03a2fa..ab097c4 100644 --- a/shared/grafana/provisioning/dashboards/llm-fleet.json +++ b/shared/grafana/provisioning/dashboards/llm-fleet.json @@ -1,7 +1,7 @@ { "uid": "llm-fleet", "title": "LLM fleet \u2014 capacity and reuse", - "description": "The signals that explain the LLM endpoints under load, from the 2026-09-16 capacity work. The panels read llama.cpp's own counters: speculative acceptance, queue depth, prefix-cache reuse, decode step rate and throughput. frank-medium (Dankserver, both R9700s, layer split) and frank-small (Dankclaw, RTX 3090) both export them; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md for the numbers these panels are read against.", + "description": "The signals that explain the LLM endpoints under load. The fleet runs two runtimes and the panels read both: vllm:* for frank-medium (Dankserver, vllm-radiance TP=2 across both R9700s since 2026-09-20) and llamacpp:* for frank-small (Dankclaw, RTX 3090) and frank-large (Dankclaw, gfx1151 iGPU \u2014 the CIRU llama.cpp build does export metrics, verified 2026-09-20). The endpoint picker selects between them. See serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md for the numbers these panels are read against, and serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split.", "tags": [ "llm", "frank-medium", @@ -42,8 +42,8 @@ { "id": 1, "type": "timeseries", - "title": "Speculative decoding (DFlash2 acceptance)", - "description": "How full each card's KV pool is. One card holds ~122,636 tokens; sustained high values with a cold prefix cache is the thrash this fleet measured.", + "title": "Prefix cache reuse rate", + "description": "rate(hits)/rate(queries) in tokens. On the llama.cpp side the prompt counter EXCLUDES cached tokens (its own HELP line), so that side divides by cached + prompt. High means pinned conversations are served from cache: measured 2026-09-20, an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused. Reuse on frank-medium happens at 1,648-token block granularity (the attention block size is padded to the mamba page size), so shared prefixes shorter than two blocks re-prefill.", "gridPos": { "h": 8, "w": 12, @@ -68,22 +68,22 @@ }, "targets": [ { - "refId": "A", - "expr": "rate(llamacpp:spec_decode_num_accepted_tokens_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total{job=~\"$job\"}[5m]), 0.001)", - "legendFormat": "{{job}} accepted per draft step" + "expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) + rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 0.001)", + "legendFormat": "{{job}} reuse (llama.cpp)", + "refId": "A" }, { - "refId": "B", - "expr": "rate(llamacpp:spec_decode_num_draft_tokens_total{job=~\"$job\"}[5m])", - "legendFormat": "{{job}} draft tokens/s" + "expr": "job:llm_vllm_prefix_cache_reuse:rate5m{job=~\"$job\"}", + "legendFormat": "{{job}} reuse (vLLM)", + "refId": "B" } ] }, { "id": 2, "type": "timeseries", - "title": "Requests waiting (queue depth)", - "description": "Anything above zero means the engine is serialising: with MAXSEQS=4 the fifth concurrent request waits.", + "title": "Speculative decoding (acceptance)", + "description": "The one failure no client can see: the drafter silently stops and the endpoint keeps answering at autoregressive speed. llama.cpp: accepted tokens per draft step. vLLM: mean accepted length (1 + accepted/drafts) from the recording rule job:llm_spec_accept_length:rate5m \u2014 measured 3.0-4.3 at boot 2026-09-20 with the DFlash2 drafter.", "gridPos": { "h": 8, "w": 12, @@ -94,35 +94,29 @@ "type": "prometheus", "uid": "prometheus" }, - "fieldConfig": { - "defaults": { - "unit": "short", - "min": 0, - "custom": { - "fillOpacity": 10, - "lineWidth": 1 - } - }, - "overrides": [] - }, "targets": [ { - "refId": "A", - "expr": "llamacpp:requests_deferred{job=~\"$job\"}", - "legendFormat": "{{job}} deferred" + "expr": "rate(llamacpp:spec_decode_num_accepted_tokens_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total{job=~\"$job\"}[5m]), 0.001)", + "legendFormat": "{{job}} accepted per draft step", + "refId": "A" }, { - "refId": "B", - "expr": "llamacpp:requests_processing{job=~\"$job\"}", - "legendFormat": "{{job}} processing" + "expr": "job:llm_spec_accept_length:rate5m{job=~\"$job\"}", + "legendFormat": "{{job}} mean accepted length (vLLM)", + "refId": "B" + }, + { + "expr": "rate(llamacpp:spec_decode_num_draft_tokens_total{job=~\"$job\"}[5m])", + "legendFormat": "{{job}} draft tokens/s", + "refId": "C" } ] }, { "id": 3, "type": "timeseries", - "title": "Prefix cache reuse rate", - "description": "rate(hits)/rate(queries). High means pinned conversations are being served from cache (measured: 22.3 s cold vs 1.4 s warm on a ~50k-token prefix). A collapse here with three concurrent large contexts is the capacity wall, not a routing failure.", + "title": "Requests waiting (queue depth)", + "description": "llama.cpp: anything above zero means the engine is serialising (with --parallel 1 a second concurrent request always shows up here). vLLM: the scheduler's own waiting/running gauges against MAXSEQS=8.", "gridPos": { "h": 8, "w": 12, @@ -133,31 +127,34 @@ "type": "prometheus", "uid": "prometheus" }, - "fieldConfig": { - "defaults": { - "unit": "percentunit", - "min": 0, - "max": 1, - "custom": { - "fillOpacity": 10, - "lineWidth": 1 - } - }, - "overrides": [] - }, "targets": [ { - "refId": "A", - "expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 1)", - "legendFormat": "{{job}} reuse" + "expr": "llamacpp:requests_deferred{job=~\"$job\"}", + "legendFormat": "{{job}} deferred", + "refId": "A" + }, + { + "expr": "llamacpp:requests_processing{job=~\"$job\"}", + "legendFormat": "{{job}} processing", + "refId": "B" + }, + { + "expr": "vllm:num_requests_waiting{job=~\"$job\"}", + "legendFormat": "{{job}} waiting (vLLM)", + "refId": "C" + }, + { + "expr": "vllm:num_requests_running{job=~\"$job\"}", + "legendFormat": "{{job}} running (vLLM)", + "refId": "D" } ] }, { "id": 4, "type": "timeseries", - "title": "Decode steps and batch size", - "description": "Wall-clock per request, prefill included \u2014 the number an agent-loop turn feels.", + "title": "KV pool fill (vLLM)", + "description": "The gauge that was retired 2026-09-16 for want of a llama.cpp equivalent and came back when frank-medium moved to vLLM: vllm:kv_cache_usage_perc (the pool is auto-sized \u2014 943,581 tokens at boot 2026-09-20). Sustained high fill with a collapsing reuse panel is the thrash; preemptions are the visible symptom of it. The llama.cpp side has no fill gauge: its capacity wall is panel 3.", "gridPos": { "h": 8, "w": 12, @@ -168,6 +165,123 @@ "type": "prometheus", "uid": "prometheus" }, + "targets": [ + { + "expr": "vllm:kv_cache_usage_perc{job=~\"$job\"}", + "legendFormat": "{{job}} KV fill %", + "refId": "A" + }, + { + "expr": "rate(vllm:num_preemptions_total{job=~\"$job\"}[5m])", + "legendFormat": "{{job}} preemptions/s", + "refId": "B" + } + ] + }, + { + "id": 5, + "type": "timeseries", + "title": "Decode and prefill throughput", + "description": "llama.cpp: llamacpp:predicted_tokens_seconds is decode, llamacpp:prompt_tokens_seconds is prompt. vLLM: the token counters, rate'd. The series carry the job label of their scrape, so the endpoint picker selects between the runtimes.", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 16 + }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "llamacpp:predicted_tokens_seconds{job=~\"$job\"}", + "legendFormat": "{{job}} decode tok/s (llama.cpp)", + "refId": "A" + }, + { + "expr": "llamacpp:prompt_tokens_seconds{job=~\"$job\"}", + "legendFormat": "{{job}} prefill tok/s (llama.cpp)", + "refId": "B" + }, + { + "expr": "rate(vllm:generation_tokens_total{job=~\"$job\"}[5m])", + "legendFormat": "{{job}} decode tok/s (vLLM)", + "refId": "C" + }, + { + "expr": "rate(vllm:prompt_tokens_total{job=~\"$job\"}[5m])", + "legendFormat": "{{job}} prefill tok/s (vLLM)", + "refId": "D" + } + ] + }, + { + "id": 6, + "type": "timeseries", + "title": "Decode steps and batch size", + "description": "llama.cpp-only signal \u2014 vLLM does not publish per-step counters. Steps per second and the busy-slot count per step (the effective batch).", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 16 + }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "rate(llamacpp:n_decode_total{job=~\"$job\"}[5m])", + "legendFormat": "{{job}} decode steps/s", + "refId": "A" + }, + { + "expr": "llamacpp:n_busy_slots_per_decode{job=~\"$job\"}", + "legendFormat": "{{job}} sequences per decode step", + "refId": "B" + } + ] + }, + { + "id": 7, + "type": "timeseries", + "title": "Firing fleet alerts", + "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 24 + }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "ALERTS{alertname=~\"Llm.*\", alertstate=\"firing\"}", + "legendFormat": "{{alertname}} {{job}}", + "refId": "A" + } + ] + }, + { + "id": 8, + "type": "timeseries", + "title": "Request latency p95 (vLLM)", + "description": "vLLM-only \u2014 the histogram the p95 panel was retired for on 2026-09-16 (llama.cpp exports no latency histogram) is back with the radiance cutover. TTFT is what a fresh prefill costs; ITL is the per-token cadence an agent loop feels; e2e is the whole request.", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 24 + }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, "fieldConfig": { "defaults": { "unit": "s", @@ -180,86 +294,19 @@ }, "targets": [ { - "refId": "A", - "expr": "rate(llamacpp:n_decode_total{job=~\"$job\"}[5m])", - "legendFormat": "{{job}} decode steps/s" + "expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:time_to_first_token_seconds_bucket{job=~\"$job\"}[5m])))", + "legendFormat": "{{job}} TTFT p95", + "refId": "A" }, { - "refId": "B", - "expr": "llamacpp:n_busy_slots_per_decode{job=~\"$job\"}", - "legendFormat": "{{job}} sequences per decode step" - } - ] - }, - { - "id": 5, - "type": "timeseries", - "title": "Decode and prefill throughput", - "description": "vLLM generation tokens, plus llama.cpp's counters for frank-small: llamacpp:predicted_tokens_seconds is decode and llamacpp:prompt_tokens_seconds is prompt throughput. The llama.cpp series carry the job label of their scrape, so the endpoint picker selects between the runtimes.", - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 16 - }, - "datasource": { - "type": "prometheus", - "uid": "prometheus" - }, - "fieldConfig": { - "defaults": { - "unit": "tokens/sec", - "custom": { - "fillOpacity": 10, - "lineWidth": 1 - } - }, - "overrides": [] - }, - "targets": [ - { - "refId": "C", - "expr": "llamacpp:predicted_tokens_seconds{job=~\"$job\"}", - "legendFormat": "{{job}} llama.cpp decode tok/s" + "expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:inter_token_latency_seconds_bucket{job=~\"$job\"}[5m])))", + "legendFormat": "{{job}} ITL p95", + "refId": "B" }, { - "refId": "D", - "expr": "llamacpp:prompt_tokens_seconds{job=~\"$job\"}", - "legendFormat": "{{job}} llama.cpp prefill tok/s" - } - ] - }, - { - "id": 6, - "type": "timeseries", - "title": "Alerts firing (ALERTS)", - "description": "Prometheus rules for the LLM fleet (shared/prometheus/rules/llm-fleet.yml): pool saturation, queueing, prefix-cache collapse, endpoint down, p95 latency. A firing alert appears here, on Prometheus' /alerts page, and nowhere else \u2014 this stack has no Alertmanager, so nothing is notified.", - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 16 - }, - "datasource": { - "type": "prometheus", - "uid": "prometheus" - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "min": 0, - "custom": { - "fillOpacity": 10, - "lineWidth": 1 - } - }, - "overrides": [] - }, - "targets": [ - { - "refId": "A", - "expr": "ALERTS{alertstate=\"firing\"}", - "legendFormat": "{{alertname}} {{job}}" + "expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket{job=~\"$job\"}[5m])))", + "legendFormat": "{{job}} e2e p95", + "refId": "C" } ] } diff --git a/shared/prometheus/rules/llm-fleet.yml b/shared/prometheus/rules/llm-fleet.yml index c43fb48..7d1f62f 100644 --- a/shared/prometheus/rules/llm-fleet.yml +++ b/shared/prometheus/rules/llm-fleet.yml @@ -16,8 +16,11 @@ # * KV pressure. vLLM exports a fill gauge (vllm:kv_cache_usage_perc) and a # preemption counter; llama.cpp exports neither, so its capacity wall is # observed as queueing (llamacpp:requests_deferred). -# * Prefix reuse. Both export a hits/queries pair, in tokens; the two -# recording rules below keep the units straight and the collapse alert +# * Prefix reuse. Both export a hits/queries pair counted in tokens, but the +# units are NOT interchangeable: llama.cpp's prompt_tokens_total EXCLUDES +# cached tokens (its own HELP line says so), so that side must divide by +# cached + prompt to get a fraction, while vLLM divides hits by queries. +# The two recording rules below keep that straight, and the collapse alert # watches both sides with one threshold. # * Speculative decoding. Both export drafts and accepted tokens, so # LlmSpecDecodeOff covers both runtimes. @@ -42,7 +45,8 @@ groups: - record: job:llm_prefix_cache_reuse:rate5m expr: | rate(llamacpp:prompt_tokens_cached_total[5m]) - / clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001) + / clamp_min(rate(llamacpp:prompt_tokens_cached_total[5m]) + + rate(llamacpp:prompt_tokens_total[5m]), 0.001) - record: job:llm_vllm_prefix_cache_reuse:rate5m expr: | @@ -112,7 +116,8 @@ groups: - alert: LlmPrefixCacheCollapse expr: >- (job:llm_prefix_cache_reuse:rate5m < 0.20 - and rate(llamacpp:prompt_tokens_total[5m]) > 0.05) + and rate(llamacpp:prompt_tokens_total[5m]) + + rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05) or (job:llm_vllm_prefix_cache_reuse:rate5m < 0.20 and rate(vllm:prefix_cache_queries_total[5m]) > 1) for: 15m @@ -162,6 +167,6 @@ groups: rootless quadlet: `systemctl --user is-active frank-medium` on Dankserver, then `podman ps -a --filter name=frank-medium`. frank-small is a root unit: `systemctl is-active frank-small` on - Dankclaw. frank-large is not scraped at all — its CIRU runtime - serves no Prometheus text endpoint — so this alert can only ever - name the endpoints that do export one. + Dankclaw. All three endpoints are scraped: frank-large's CIRU + llama.cpp build does export the llamacpp:* family (verified + 2026-09-20), so this alert covers the whole fleet.