dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity

Browse Source 
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
  divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
  cached tokens (its own HELP line), so the ratio was unbounded — measured
  77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
  fire on that side. Now cached / (cached + prompt), with the traffic floor
  counting both terms too; the header documents the unit trap. Also fixed the
  LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
  the llamacpp:* family (verified live), so the alert covers all three
  endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
  namespaces. Every panel reads both (the endpoint picker selects); the
  KV-pool-fill panel retired 2026-09-16 returned with
  vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
  histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
  Llm.* rules; the stale halogen-era description is gone. All 21 panel
  expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
  reuse happens at 1,648-token block granularity (attention block size padded
  to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
  to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
  registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.

./tests/run.sh: 325 pass.
This commit is contained in:
Frank
2026-09-20 19:53:14 -07:00
parent 08f1c7ad09
commit c42780f9f3
5 changed files with 220 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files
shared/grafana/provisioning/dashboards/llm-fleet.json
+170 -123
View File
@@ -1,7 +1,7 @@
{
"uid": "llm-fleet",
"title": "LLM fleet \u2014 capacity and reuse",
"description": "The signals that explain the LLM endpoints under load, from the 2026-09-16 capacity work. The panels read llama.cpp's own counters: speculative acceptance, queue depth, prefix-cache reuse, decode step rate and throughput. frank-medium (Dankserver, both R9700s, layer split) and frank-small (Dankclaw, RTX 3090) both export them; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md for the numbers these panels are read against.",
"description": "The signals that explain the LLM endpoints under load. The fleet runs two runtimes and the panels read both: vllm:* for frank-medium (Dankserver, vllm-radiance TP=2 across both R9700s since 2026-09-20) and llamacpp:* for frank-small (Dankclaw, RTX 3090) and frank-large (Dankclaw, gfx1151 iGPU \u2014 the CIRU llama.cpp build does export metrics, verified 2026-09-20). The endpoint picker selects between them. See serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md for the numbers these panels are read against, and serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split.",
"tags": [
"llm",
"frank-medium",
@@ -42,8 +42,8 @@
{
"id": 1,
"type": "timeseries",
"title": "Speculative decoding (DFlash2 acceptance)",
"description": "How full each card's KV pool is. One card holds ~122,636 tokens; sustained high values with a cold prefix cache is the thrash this fleet measured.",
"title": "Prefix cache reuse rate",
"description": "rate(hits)/rate(queries) in tokens. On the llama.cpp side the prompt counter EXCLUDES cached tokens (its own HELP line), so that side divides by cached + prompt. High means pinned conversations are served from cache: measured 2026-09-20, an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused. Reuse on frank-medium happens at 1,648-token block granularity (the attention block size is padded to the mamba page size), so shared prefixes shorter than two blocks re-prefill.",
"gridPos": {
"h": 8,
"w": 12,
@@ -68,22 +68,22 @@
},
"targets": [
{
"refId": "A",
"expr": "rate(llamacpp:spec_decode_num_accepted_tokens_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total{job=~\"$job\"}[5m]), 0.001)",
"legendFormat": "{{job}} accepted per draft step"
"expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) + rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 0.001)",
"legendFormat": "{{job}} reuse (llama.cpp)",
"refId": "A"
},
{
"refId": "B",
"expr": "rate(llamacpp:spec_decode_num_draft_tokens_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} draft tokens/s"
"expr": "job:llm_vllm_prefix_cache_reuse:rate5m{job=~\"$job\"}",
"legendFormat": "{{job}} reuse (vLLM)",
"refId": "B"
}
]
},
{
"id": 2,
"type": "timeseries",
"title": "Requests waiting (queue depth)",
"description": "Anything above zero means the engine is serialising: with MAXSEQS=4 the fifth concurrent request waits.",
"title": "Speculative decoding (acceptance)",
"description": "The one failure no client can see: the drafter silently stops and the endpoint keeps answering at autoregressive speed. llama.cpp: accepted tokens per draft step. vLLM: mean accepted length (1 + accepted/drafts) from the recording rule job:llm_spec_accept_length:rate5m \u2014 measured 3.0-4.3 at boot 2026-09-20 with the DFlash2 drafter.",
"gridPos": {
"h": 8,
"w": 12,
@@ -94,35 +94,29 @@
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0,
"custom": {
"fillOpacity": 10,
"lineWidth": 1
}
},
"overrides": []
},
"targets": [
{
"refId": "A",
"expr": "llamacpp:requests_deferred{job=~\"$job\"}",
"legendFormat": "{{job}} deferred"
"expr": "rate(llamacpp:spec_decode_num_accepted_tokens_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total{job=~\"$job\"}[5m]), 0.001)",
"legendFormat": "{{job}} accepted per draft step",
"refId": "A"
},
{
"refId": "B",
"expr": "llamacpp:requests_processing{job=~\"$job\"}",
"legendFormat": "{{job}} processing"
"expr": "job:llm_spec_accept_length:rate5m{job=~\"$job\"}",
"legendFormat": "{{job}} mean accepted length (vLLM)",
"refId": "B"
},
{
"expr": "rate(llamacpp:spec_decode_num_draft_tokens_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} draft tokens/s",
"refId": "C"
}
]
},
{
"id": 3,
"type": "timeseries",
"title": "Prefix cache reuse rate",
"description": "rate(hits)/rate(queries). High means pinned conversations are being served from cache (measured: 22.3 s cold vs 1.4 s warm on a ~50k-token prefix). A collapse here with three concurrent large contexts is the capacity wall, not a routing failure.",
"title": "Requests waiting (queue depth)",
"description": "llama.cpp: anything above zero means the engine is serialising (with --parallel 1 a second concurrent request always shows up here). vLLM: the scheduler's own waiting/running gauges against MAXSEQS=8.",
"gridPos": {
"h": 8,
"w": 12,
@@ -133,31 +127,34 @@
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1,
"custom": {
"fillOpacity": 10,
"lineWidth": 1
}
},
"overrides": []
},
"targets": [
{
"refId": "A",
"expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 1)",
"legendFormat": "{{job}} reuse"
"expr": "llamacpp:requests_deferred{job=~\"$job\"}",
"legendFormat": "{{job}} deferred",
"refId": "A"
},
{
"expr": "llamacpp:requests_processing{job=~\"$job\"}",
"legendFormat": "{{job}} processing",
"refId": "B"
},
{
"expr": "vllm:num_requests_waiting{job=~\"$job\"}",
"legendFormat": "{{job}} waiting (vLLM)",
"refId": "C"
},
{
"expr": "vllm:num_requests_running{job=~\"$job\"}",
"legendFormat": "{{job}} running (vLLM)",
"refId": "D"
}
]
},
{
"id": 4,
"type": "timeseries",
"title": "Decode steps and batch size",
"description": "Wall-clock per request, prefill included \u2014 the number an agent-loop turn feels.",
"title": "KV pool fill (vLLM)",
"description": "The gauge that was retired 2026-09-16 for want of a llama.cpp equivalent and came back when frank-medium moved to vLLM: vllm:kv_cache_usage_perc (the pool is auto-sized \u2014 943,581 tokens at boot 2026-09-20). Sustained high fill with a collapsing reuse panel is the thrash; preemptions are the visible symptom of it. The llama.cpp side has no fill gauge: its capacity wall is panel 3.",
"gridPos": {
"h": 8,
"w": 12,
@@ -168,6 +165,123 @@
"type": "prometheus",
"uid": "prometheus"
},
"targets": [
{
"expr": "vllm:kv_cache_usage_perc{job=~\"$job\"}",
"legendFormat": "{{job}} KV fill %",
"refId": "A"
},
{
"expr": "rate(vllm:num_preemptions_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} preemptions/s",
"refId": "B"
}
]
},
{
"id": 5,
"type": "timeseries",
"title": "Decode and prefill throughput",
"description": "llama.cpp: llamacpp:predicted_tokens_seconds is decode, llamacpp:prompt_tokens_seconds is prompt. vLLM: the token counters, rate'd. The series carry the job label of their scrape, so the endpoint picker selects between the runtimes.",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"targets": [
{
"expr": "llamacpp:predicted_tokens_seconds{job=~\"$job\"}",
"legendFormat": "{{job}} decode tok/s (llama.cpp)",
"refId": "A"
},
{
"expr": "llamacpp:prompt_tokens_seconds{job=~\"$job\"}",
"legendFormat": "{{job}} prefill tok/s (llama.cpp)",
"refId": "B"
},
{
"expr": "rate(vllm:generation_tokens_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} decode tok/s (vLLM)",
"refId": "C"
},
{
"expr": "rate(vllm:prompt_tokens_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} prefill tok/s (vLLM)",
"refId": "D"
}
]
},
{
"id": 6,
"type": "timeseries",
"title": "Decode steps and batch size",
"description": "llama.cpp-only signal \u2014 vLLM does not publish per-step counters. Steps per second and the busy-slot count per step (the effective batch).",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"targets": [
{
"expr": "rate(llamacpp:n_decode_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} decode steps/s",
"refId": "A"
},
{
"expr": "llamacpp:n_busy_slots_per_decode{job=~\"$job\"}",
"legendFormat": "{{job}} sequences per decode step",
"refId": "B"
}
]
},
{
"id": 7,
"type": "timeseries",
"title": "Firing fleet alerts",
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 24
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"targets": [
{
"expr": "ALERTS{alertname=~\"Llm.*\", alertstate=\"firing\"}",
"legendFormat": "{{alertname}} {{job}}",
"refId": "A"
}
]
},
{
"id": 8,
"type": "timeseries",
"title": "Request latency p95 (vLLM)",
"description": "vLLM-only \u2014 the histogram the p95 panel was retired for on 2026-09-16 (llama.cpp exports no latency histogram) is back with the radiance cutover. TTFT is what a fresh prefill costs; ITL is the per-token cadence an agent loop feels; e2e is the whole request.",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 24
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "s",
@@ -180,86 +294,19 @@
},
"targets": [
{
"refId": "A",
"expr": "rate(llamacpp:n_decode_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} decode steps/s"
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:time_to_first_token_seconds_bucket{job=~\"$job\"}[5m])))",
"legendFormat": "{{job}} TTFT p95",
"refId": "A"
},
{
"refId": "B",
"expr": "llamacpp:n_busy_slots_per_decode{job=~\"$job\"}",
"legendFormat": "{{job}} sequences per decode step"
}
]
},
{
"id": 5,
"type": "timeseries",
"title": "Decode and prefill throughput",
"description": "vLLM generation tokens, plus llama.cpp's counters for frank-small: llamacpp:predicted_tokens_seconds is decode and llamacpp:prompt_tokens_seconds is prompt throughput. The llama.cpp series carry the job label of their scrape, so the endpoint picker selects between the runtimes.",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "tokens/sec",
"custom": {
"fillOpacity": 10,
"lineWidth": 1
}
},
"overrides": []
},
"targets": [
{
"refId": "C",
"expr": "llamacpp:predicted_tokens_seconds{job=~\"$job\"}",
"legendFormat": "{{job}} llama.cpp decode tok/s"
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:inter_token_latency_seconds_bucket{job=~\"$job\"}[5m])))",
"legendFormat": "{{job}} ITL p95",
"refId": "B"
},
{
"refId": "D",
"expr": "llamacpp:prompt_tokens_seconds{job=~\"$job\"}",
"legendFormat": "{{job}} llama.cpp prefill tok/s"
}
]
},
{
"id": 6,
"type": "timeseries",
"title": "Alerts firing (ALERTS)",
"description": "Prometheus rules for the LLM fleet (shared/prometheus/rules/llm-fleet.yml): pool saturation, queueing, prefix-cache collapse, endpoint down, p95 latency. A firing alert appears here, on Prometheus' /alerts page, and nowhere else \u2014 this stack has no Alertmanager, so nothing is notified.",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0,
"custom": {
"fillOpacity": 10,
"lineWidth": 1
}
},
"overrides": []
},
"targets": [
{
"refId": "A",
"expr": "ALERTS{alertstate=\"firing\"}",
"legendFormat": "{{alertname}} {{job}}"
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket{job=~\"$job\"}[5m])))",
"legendFormat": "{{job}} e2e p95",
"refId": "C"
}
]
}
shared/prometheus/rules/llm-fleet.yml
+12 -7
View File
@@ -16,8 +16,11 @@
# * KV pressure. vLLM exports a fill gauge (vllm:kv_cache_usage_perc) and a
# preemption counter; llama.cpp exports neither, so its capacity wall is
# observed as queueing (llamacpp:requests_deferred).
# * Prefix reuse. Both export a hits/queries pair, in tokens; the two
# recording rules below keep the units straight and the collapse alert
# * Prefix reuse. Both export a hits/queries pair counted in tokens, but the
# units are NOT interchangeable: llama.cpp's prompt_tokens_total EXCLUDES
# cached tokens (its own HELP line says so), so that side must divide by
# cached + prompt to get a fraction, while vLLM divides hits by queries.
# The two recording rules below keep that straight, and the collapse alert
# watches both sides with one threshold.
# * Speculative decoding. Both export drafts and accepted tokens, so
# LlmSpecDecodeOff covers both runtimes.
@@ -42,7 +45,8 @@ groups:
- record: job:llm_prefix_cache_reuse:rate5m
expr: |
rate(llamacpp:prompt_tokens_cached_total[5m])
/ clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001)
/ clamp_min(rate(llamacpp:prompt_tokens_cached_total[5m])
+ rate(llamacpp:prompt_tokens_total[5m]), 0.001)
- record: job:llm_vllm_prefix_cache_reuse:rate5m
expr: |
@@ -112,7 +116,8 @@ groups:
- alert: LlmPrefixCacheCollapse
expr: >-
(job:llm_prefix_cache_reuse:rate5m < 0.20
and rate(llamacpp:prompt_tokens_total[5m]) > 0.05)
and rate(llamacpp:prompt_tokens_total[5m])
+ rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05)
or (job:llm_vllm_prefix_cache_reuse:rate5m < 0.20
and rate(vllm:prefix_cache_queries_total[5m]) > 1)
for: 15m
@@ -162,6 +167,6 @@ groups:
rootless quadlet: `systemctl --user is-active frank-medium` on
Dankserver, then `podman ps -a --filter name=frank-medium`.
frank-small is a root unit: `systemctl is-active frank-small` on
Dankclaw. frank-large is not scraped at all — its CIRU runtime
serves no Prometheus text endpoint — so this alert can only ever
name the endpoints that do export one.
Dankclaw. All three endpoints are scraped: frank-large's CIRU
llama.cpp build does export the llamacpp:* family (verified
2026-09-20), so this alert covers the whole fleet.