fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES cached tokens (its own HELP line), so the ratio was unbounded — measured 77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never fire on that side. Now cached / (cached + prompt), with the traffic floor counting both terms too; the header documents the unit trap. Also fixed the LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export the llamacpp:* family (verified live), so the alert covers all three endpoints. - shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two namespaces. Every panel reads both (the endpoint picker selects); the KV-pool-fill panel retired 2026-09-16 returned with vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's Llm.* rules; the stale halogen-era description is gone. All 21 panel expressions validated against the live endpoints. - runbook + log + Observability: prefix caching measured, not inferred — reuse happens at 1,648-token block granularity (attention block size padded to the mamba page size); an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules. ./tests/run.sh: 325 pass.
This commit is contained in:
Frank
@@ -41,12 +41,12 @@ Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in
|
||||
- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
|
||||
- **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
|
||||
- **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the relay binds the host's LAN IP (192.168.0.143:18093, the one address netavark delivers that dial to), with an iptables set admitting loopback, the host and the podman bridge subnet only. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
|
||||
- **2026-09-20: the rule file now spans two metric namespaces.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — seven alerts and two recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. **The provisioned dashboard is the known gap:** its panels are still keyed to `llamacpp:*` series, so frank-medium's row went blank at the cutover while frank-small's and frank-large's still draw. Re-cut pending — the KV-fill and latency panels that were retired for llama.cpp have vLLM equivalents again (`vllm:kv_cache_usage_perc`, `vllm:inter_token_latency_seconds`).
|
||||
- **2026-09-20: the rule file now spans two metric namespaces, and the dashboard was re-cut the same day.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — six alerts and three recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. One semantics trap, measured 2026-09-20: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP line), so the llama.cpp reuse rule divides by `cached + prompt` — the naive `cached/prompt` ratio reads 77 on a fully-warm endpoint and can never show a collapse. The provisioned dashboard's panels now read both namespaces (the endpoint picker selects between them); the KV-pool-fill panel retired 2026-09-16 returned with `vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every expression in the rules and the dashboard was validated against the live endpoints on 2026-09-20.
|
||||
- **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
|
||||
- **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
|
||||
- **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
|
||||
|
||||
The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, five rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent.
|
||||
The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, nine rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent.
|
||||
|
||||
## Deployment
|
||||
|
||||
|
||||
@@ -68,6 +68,27 @@ against the host's 62 GiB and stayed flat through a 4-way burst at 512-token
|
||||
max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the
|
||||
two-replica vision shape, not this one.
|
||||
|
||||
## Prefix caching — measured granularity (2026-09-20)
|
||||
|
||||
`--enable-prefix-caching` is on, and radiance says so at boot: *"Prefix caching
|
||||
in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers
|
||||
is experimental."* The attention block size is **1,648 tokens** (boot log:
|
||||
"Setting attention block size to 1648 tokens …"), padded to the mamba/GDN page
|
||||
size, and reuse happens at that granularity — not per token.
|
||||
|
||||
Measured on the deployed endpoint, not inferred:
|
||||
|
||||
| Probe | Cold | Warm | `vllm:prefix_cache_hits_total` delta |
|
||||
|---|---|---|---|
|
||||
| identical 13,021-token prompt | 2.30 s | 0.68 s (3.4×) | +9,888 of 13,021 queried (76 %) |
|
||||
| identical 2,521-token prompt | 0.51 s | 0.51 s | +0 (under two blocks) |
|
||||
|
||||
Consequence: a shared prefix shorter than about two blocks (≈3.3k tokens)
|
||||
re-prefills in full, so the benefit starts at ~3.3k tokens of shared prefix.
|
||||
The API returns `prompt_tokens_details: null` in this build, so the
|
||||
`vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total` counters
|
||||
(both token-counted) are the only instrument — see [[Observability]].
|
||||
|
||||
## Battery results — why this shape won (2026-09-20)
|
||||
|
||||
`scripts/bench-conversation.py`, 10 real-workload prompts, `max_tokens=512`,
|
||||
|
||||
@@ -46,14 +46,24 @@ timestamp: 2026-09-14
|
||||
≤ 262,144).
|
||||
- **Monitoring cut over with the runtime.** `shared/prometheus/rules/llm-fleet.yml`
|
||||
now spans two namespaces — `vllm:*` for frank-medium, `llamacpp:*` for the
|
||||
Dankclaw pair — with 7 alerts + 2 recording rules; KV-saturation and preemption
|
||||
Dankclaw pair — with 6 alerts + 3 recording rules; KV-saturation and preemption
|
||||
pressure returned (vLLM exports both), and the two engine-independent rules OR
|
||||
the families with per-family traffic floors. Every expression validated against
|
||||
live Prometheus. **Two scrape targets were red by construction and are fixed**:
|
||||
the families with per-family traffic floors. One semantics trap fixed the same
|
||||
day: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP
|
||||
line), so the reuse rule divides by `cached + prompt` — the naive
|
||||
`cached/prompt` reads 77 on a fully-warm endpoint and can never show a
|
||||
collapse. **Two scrape targets were red by construction and are fixed**:
|
||||
frank-small and frank-large are *host processes* on Dankclaw and netavark DNATs
|
||||
only published ports, so both are now addressed as `host.containers.internal`.
|
||||
Known gap: the provisioned Grafana dashboard is still keyed to `llamacpp:*`, so
|
||||
frank-medium's panels are blank pending a re-cut.
|
||||
The provisioned Grafana dashboard was re-cut the same day: every panel reads
|
||||
both namespaces (the endpoint picker selects), the KV-fill panel returned with
|
||||
`vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM
|
||||
histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every
|
||||
rule and dashboard expression validated against the live endpoints. **Prefix
|
||||
caching measured, not inferred**: reuse happens at 1,648-token block granularity
|
||||
(the attention block size is padded to the mamba page size) — an identical
|
||||
13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %)
|
||||
reused, while a 2,521-token prompt registered zero hits (under two blocks).
|
||||
- **Network decision.** `eno2` (5GbE, 192.168.0.247) kept, `eno1` pinned primary
|
||||
via metric 100 vs 200 in `scripts/dankserver-route-primary.sh`; applied with
|
||||
`nmcli device reapply` (there is no `nmcli connection reapply` in NM 1.54).
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"uid": "llm-fleet",
|
||||
"title": "LLM fleet \u2014 capacity and reuse",
|
||||
"description": "The signals that explain the LLM endpoints under load, from the 2026-09-16 capacity work. The panels read llama.cpp's own counters: speculative acceptance, queue depth, prefix-cache reuse, decode step rate and throughput. frank-medium (Dankserver, both R9700s, layer split) and frank-small (Dankclaw, RTX 3090) both export them; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md for the numbers these panels are read against.",
|
||||
"description": "The signals that explain the LLM endpoints under load. The fleet runs two runtimes and the panels read both: vllm:* for frank-medium (Dankserver, vllm-radiance TP=2 across both R9700s since 2026-09-20) and llamacpp:* for frank-small (Dankclaw, RTX 3090) and frank-large (Dankclaw, gfx1151 iGPU \u2014 the CIRU llama.cpp build does export metrics, verified 2026-09-20). The endpoint picker selects between them. See serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md for the numbers these panels are read against, and serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split.",
|
||||
"tags": [
|
||||
"llm",
|
||||
"frank-medium",
|
||||
@@ -42,8 +42,8 @@
|
||||
{
|
||||
"id": 1,
|
||||
"type": "timeseries",
|
||||
"title": "Speculative decoding (DFlash2 acceptance)",
|
||||
"description": "How full each card's KV pool is. One card holds ~122,636 tokens; sustained high values with a cold prefix cache is the thrash this fleet measured.",
|
||||
"title": "Prefix cache reuse rate",
|
||||
"description": "rate(hits)/rate(queries) in tokens. On the llama.cpp side the prompt counter EXCLUDES cached tokens (its own HELP line), so that side divides by cached + prompt. High means pinned conversations are served from cache: measured 2026-09-20, an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused. Reuse on frank-medium happens at 1,648-token block granularity (the attention block size is padded to the mamba page size), so shared prefixes shorter than two blocks re-prefill.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
@@ -68,22 +68,22 @@
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(llamacpp:spec_decode_num_accepted_tokens_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total{job=~\"$job\"}[5m]), 0.001)",
|
||||
"legendFormat": "{{job}} accepted per draft step"
|
||||
"expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) + rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 0.001)",
|
||||
"legendFormat": "{{job}} reuse (llama.cpp)",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "rate(llamacpp:spec_decode_num_draft_tokens_total{job=~\"$job\"}[5m])",
|
||||
"legendFormat": "{{job}} draft tokens/s"
|
||||
"expr": "job:llm_vllm_prefix_cache_reuse:rate5m{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} reuse (vLLM)",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "timeseries",
|
||||
"title": "Requests waiting (queue depth)",
|
||||
"description": "Anything above zero means the engine is serialising: with MAXSEQS=4 the fifth concurrent request waits.",
|
||||
"title": "Speculative decoding (acceptance)",
|
||||
"description": "The one failure no client can see: the drafter silently stops and the endpoint keeps answering at autoregressive speed. llama.cpp: accepted tokens per draft step. vLLM: mean accepted length (1 + accepted/drafts) from the recording rule job:llm_spec_accept_length:rate5m \u2014 measured 3.0-4.3 at boot 2026-09-20 with the DFlash2 drafter.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
@@ -94,35 +94,29 @@
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0,
|
||||
"custom": {
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "llamacpp:requests_deferred{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} deferred"
|
||||
"expr": "rate(llamacpp:spec_decode_num_accepted_tokens_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total{job=~\"$job\"}[5m]), 0.001)",
|
||||
"legendFormat": "{{job}} accepted per draft step",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "llamacpp:requests_processing{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} processing"
|
||||
"expr": "job:llm_spec_accept_length:rate5m{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} mean accepted length (vLLM)",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "rate(llamacpp:spec_decode_num_draft_tokens_total{job=~\"$job\"}[5m])",
|
||||
"legendFormat": "{{job}} draft tokens/s",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "timeseries",
|
||||
"title": "Prefix cache reuse rate",
|
||||
"description": "rate(hits)/rate(queries). High means pinned conversations are being served from cache (measured: 22.3 s cold vs 1.4 s warm on a ~50k-token prefix). A collapse here with three concurrent large contexts is the capacity wall, not a routing failure.",
|
||||
"title": "Requests waiting (queue depth)",
|
||||
"description": "llama.cpp: anything above zero means the engine is serialising (with --parallel 1 a second concurrent request always shows up here). vLLM: the scheduler's own waiting/running gauges against MAXSEQS=8.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
@@ -133,31 +127,34 @@
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1,
|
||||
"custom": {
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 1)",
|
||||
"legendFormat": "{{job}} reuse"
|
||||
"expr": "llamacpp:requests_deferred{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} deferred",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "llamacpp:requests_processing{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} processing",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "vllm:num_requests_waiting{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} waiting (vLLM)",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"expr": "vllm:num_requests_running{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} running (vLLM)",
|
||||
"refId": "D"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "timeseries",
|
||||
"title": "Decode steps and batch size",
|
||||
"description": "Wall-clock per request, prefill included \u2014 the number an agent-loop turn feels.",
|
||||
"title": "KV pool fill (vLLM)",
|
||||
"description": "The gauge that was retired 2026-09-16 for want of a llama.cpp equivalent and came back when frank-medium moved to vLLM: vllm:kv_cache_usage_perc (the pool is auto-sized \u2014 943,581 tokens at boot 2026-09-20). Sustained high fill with a collapsing reuse panel is the thrash; preemptions are the visible symptom of it. The llama.cpp side has no fill gauge: its capacity wall is panel 3.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
@@ -168,6 +165,123 @@
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "vllm:kv_cache_usage_perc{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} KV fill %",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "rate(vllm:num_preemptions_total{job=~\"$job\"}[5m])",
|
||||
"legendFormat": "{{job}} preemptions/s",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Decode and prefill throughput",
|
||||
"description": "llama.cpp: llamacpp:predicted_tokens_seconds is decode, llamacpp:prompt_tokens_seconds is prompt. vLLM: the token counters, rate'd. The series carry the job label of their scrape, so the endpoint picker selects between the runtimes.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llamacpp:predicted_tokens_seconds{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} decode tok/s (llama.cpp)",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "llamacpp:prompt_tokens_seconds{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} prefill tok/s (llama.cpp)",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "rate(vllm:generation_tokens_total{job=~\"$job\"}[5m])",
|
||||
"legendFormat": "{{job}} decode tok/s (vLLM)",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"expr": "rate(vllm:prompt_tokens_total{job=~\"$job\"}[5m])",
|
||||
"legendFormat": "{{job}} prefill tok/s (vLLM)",
|
||||
"refId": "D"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Decode steps and batch size",
|
||||
"description": "llama.cpp-only signal \u2014 vLLM does not publish per-step counters. Steps per second and the busy-slot count per step (the effective batch).",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(llamacpp:n_decode_total{job=~\"$job\"}[5m])",
|
||||
"legendFormat": "{{job}} decode steps/s",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "llamacpp:n_busy_slots_per_decode{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} sequences per decode step",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "timeseries",
|
||||
"title": "Firing fleet alerts",
|
||||
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "ALERTS{alertname=~\"Llm.*\", alertstate=\"firing\"}",
|
||||
"legendFormat": "{{alertname}} {{job}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "timeseries",
|
||||
"title": "Request latency p95 (vLLM)",
|
||||
"description": "vLLM-only \u2014 the histogram the p95 panel was retired for on 2026-09-16 (llama.cpp exports no latency histogram) is back with the radiance cutover. TTFT is what a fresh prefill costs; ITL is the per-token cadence an agent loop feels; e2e is the whole request.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 24
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
@@ -180,86 +294,19 @@
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(llamacpp:n_decode_total{job=~\"$job\"}[5m])",
|
||||
"legendFormat": "{{job}} decode steps/s"
|
||||
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:time_to_first_token_seconds_bucket{job=~\"$job\"}[5m])))",
|
||||
"legendFormat": "{{job}} TTFT p95",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "llamacpp:n_busy_slots_per_decode{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} sequences per decode step"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Decode and prefill throughput",
|
||||
"description": "vLLM generation tokens, plus llama.cpp's counters for frank-small: llamacpp:predicted_tokens_seconds is decode and llamacpp:prompt_tokens_seconds is prompt throughput. The llama.cpp series carry the job label of their scrape, so the endpoint picker selects between the runtimes.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "tokens/sec",
|
||||
"custom": {
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "llamacpp:predicted_tokens_seconds{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} llama.cpp decode tok/s"
|
||||
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:inter_token_latency_seconds_bucket{job=~\"$job\"}[5m])))",
|
||||
"legendFormat": "{{job}} ITL p95",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"refId": "D",
|
||||
"expr": "llamacpp:prompt_tokens_seconds{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} llama.cpp prefill tok/s"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Alerts firing (ALERTS)",
|
||||
"description": "Prometheus rules for the LLM fleet (shared/prometheus/rules/llm-fleet.yml): pool saturation, queueing, prefix-cache collapse, endpoint down, p95 latency. A firing alert appears here, on Prometheus' /alerts page, and nowhere else \u2014 this stack has no Alertmanager, so nothing is notified.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0,
|
||||
"custom": {
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "ALERTS{alertstate=\"firing\"}",
|
||||
"legendFormat": "{{alertname}} {{job}}"
|
||||
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket{job=~\"$job\"}[5m])))",
|
||||
"legendFormat": "{{job}} e2e p95",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -16,8 +16,11 @@
|
||||
# * KV pressure. vLLM exports a fill gauge (vllm:kv_cache_usage_perc) and a
|
||||
# preemption counter; llama.cpp exports neither, so its capacity wall is
|
||||
# observed as queueing (llamacpp:requests_deferred).
|
||||
# * Prefix reuse. Both export a hits/queries pair, in tokens; the two
|
||||
# recording rules below keep the units straight and the collapse alert
|
||||
# * Prefix reuse. Both export a hits/queries pair counted in tokens, but the
|
||||
# units are NOT interchangeable: llama.cpp's prompt_tokens_total EXCLUDES
|
||||
# cached tokens (its own HELP line says so), so that side must divide by
|
||||
# cached + prompt to get a fraction, while vLLM divides hits by queries.
|
||||
# The two recording rules below keep that straight, and the collapse alert
|
||||
# watches both sides with one threshold.
|
||||
# * Speculative decoding. Both export drafts and accepted tokens, so
|
||||
# LlmSpecDecodeOff covers both runtimes.
|
||||
@@ -42,7 +45,8 @@ groups:
|
||||
- record: job:llm_prefix_cache_reuse:rate5m
|
||||
expr: |
|
||||
rate(llamacpp:prompt_tokens_cached_total[5m])
|
||||
/ clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001)
|
||||
/ clamp_min(rate(llamacpp:prompt_tokens_cached_total[5m])
|
||||
+ rate(llamacpp:prompt_tokens_total[5m]), 0.001)
|
||||
|
||||
- record: job:llm_vllm_prefix_cache_reuse:rate5m
|
||||
expr: |
|
||||
@@ -112,7 +116,8 @@ groups:
|
||||
- alert: LlmPrefixCacheCollapse
|
||||
expr: >-
|
||||
(job:llm_prefix_cache_reuse:rate5m < 0.20
|
||||
and rate(llamacpp:prompt_tokens_total[5m]) > 0.05)
|
||||
and rate(llamacpp:prompt_tokens_total[5m])
|
||||
+ rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05)
|
||||
or (job:llm_vllm_prefix_cache_reuse:rate5m < 0.20
|
||||
and rate(vllm:prefix_cache_queries_total[5m]) > 1)
|
||||
for: 15m
|
||||
@@ -162,6 +167,6 @@ groups:
|
||||
rootless quadlet: `systemctl --user is-active frank-medium` on
|
||||
Dankserver, then `podman ps -a --filter name=frank-medium`.
|
||||
frank-small is a root unit: `systemctl is-active frank-small` on
|
||||
Dankclaw. frank-large is not scraped at all — its CIRU runtime
|
||||
serves no Prometheus text endpoint — so this alert can only ever
|
||||
name the endpoints that do export one.
|
||||
Dankclaw. All three endpoints are scraped: frank-large's CIRU
|
||||
llama.cpp build does export the llamacpp:* family (verified
|
||||
2026-09-20), so this alert covers the whole fleet.
|
||||
|
||||
Reference in New Issue
Block a user