{
  "uid": "llm-fleet",
  "title": "LLM fleet \u2014 capacity and reuse",
  "description": "The signals that explain the LLM endpoints under load. The fleet runs two runtimes and the panels read both: vllm:* for frank-medium (Dankserver, vllm-radiance TP=2 across both R9700s since 2026-09-20) and llamacpp:* for frank-small (Dankclaw, RTX 3090) and frank-large (Dankclaw, gfx1151 iGPU \u2014 the CIRU llama.cpp build does export metrics, verified 2026-09-20). The endpoint picker selects between them. See serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md for the numbers these panels are read against, and serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split.",
  "tags": [
    "llm",
    "frank-medium",
    "capacity"
  ],
  "timezone": "browser",
  "schemaVersion": 39,
  "version": 1,
  "refresh": "30s",
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "templating": {
    "list": [
      {
        "name": "job",
        "label": "endpoint",
        "type": "query",
        "datasource": {
          "type": "prometheus",
          "uid": "prometheus"
        },
        "query": {
          "query": "label_values({__name__=~\"vllm:kv_cache_usage_perc|llamacpp:.*\"}, job)",
          "refId": "A"
        },
        "includeAll": true,
        "multi": true,
        "current": {
          "text": "All",
          "value": "$__all"
        }
      }
    ]
  },
  "panels": [
    {
      "id": 1,
      "type": "timeseries",
      "title": "Prefix cache reuse rate",
      "description": "rate(hits)/rate(queries) in tokens. On the llama.cpp side the prompt counter EXCLUDES cached tokens (its own HELP line), so that side divides by cached + prompt. High means pinned conversations are served from cache: measured 2026-09-20, an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused. Reuse on frank-medium happens at 1,648-token block granularity (the attention block size is padded to the mamba page size), so shared prefixes shorter than two blocks re-prefill.",
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "min": 0,
          "max": 1,
          "custom": {
            "fillOpacity": 10,
            "lineWidth": 1
          }
        },
        "overrides": []
      },
      "targets": [
        {
          "expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) + rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 0.001)",
          "legendFormat": "{{job}} reuse (llama.cpp)",
          "refId": "A"
        },
        {
          "expr": "job:llm_vllm_prefix_cache_reuse:rate5m{job=~\"$job\"}",
          "legendFormat": "{{job}} reuse (vLLM)",
          "refId": "B"
        }
      ]
    },
    {
      "id": 2,
      "type": "timeseries",
      "title": "Speculative decoding (acceptance)",
      "description": "The one failure no client can see: the drafter silently stops and the endpoint keeps answering at autoregressive speed. llama.cpp: accepted tokens per draft step. vLLM: mean accepted length (1 + accepted/drafts) from the recording rule job:llm_spec_accept_length:rate5m \u2014 measured 3.0-4.3 at boot 2026-09-20 with the DFlash2 drafter.",
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 0
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "targets": [
        {
          "expr": "rate(llamacpp:spec_decode_num_accepted_tokens_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total{job=~\"$job\"}[5m]), 0.001)",
          "legendFormat": "{{job}} accepted per draft step",
          "refId": "A"
        },
        {
          "expr": "job:llm_spec_accept_length:rate5m{job=~\"$job\"}",
          "legendFormat": "{{job}} mean accepted length (vLLM)",
          "refId": "B"
        },
        {
          "expr": "rate(llamacpp:spec_decode_num_draft_tokens_total{job=~\"$job\"}[5m])",
          "legendFormat": "{{job}} draft tokens/s",
          "refId": "C"
        }
      ]
    },
    {
      "id": 3,
      "type": "timeseries",
      "title": "Requests waiting (queue depth)",
      "description": "llama.cpp: anything above zero means the engine is serialising (with --parallel 1 a second concurrent request always shows up here). vLLM: the scheduler's own waiting/running gauges against MAXSEQS=8.",
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 8
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "targets": [
        {
          "expr": "llamacpp:requests_deferred{job=~\"$job\"}",
          "legendFormat": "{{job}} deferred",
          "refId": "A"
        },
        {
          "expr": "llamacpp:requests_processing{job=~\"$job\"}",
          "legendFormat": "{{job}} processing",
          "refId": "B"
        },
        {
          "expr": "vllm:num_requests_waiting{job=~\"$job\"}",
          "legendFormat": "{{job}} waiting (vLLM)",
          "refId": "C"
        },
        {
          "expr": "vllm:num_requests_running{job=~\"$job\"}",
          "legendFormat": "{{job}} running (vLLM)",
          "refId": "D"
        }
      ]
    },
    {
      "id": 4,
      "type": "timeseries",
      "title": "KV pool fill (vLLM)",
      "description": "The gauge that was retired 2026-09-16 for want of a llama.cpp equivalent and came back when frank-medium moved to vLLM: vllm:kv_cache_usage_perc (the pool is auto-sized \u2014 943,581 tokens at boot 2026-09-20). Sustained high fill with a collapsing reuse panel is the thrash; preemptions are the visible symptom of it. The llama.cpp side has no fill gauge: its capacity wall is panel 3.",
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 8
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "targets": [
        {
          "expr": "vllm:kv_cache_usage_perc{job=~\"$job\"}",
          "legendFormat": "{{job}} KV fill %",
          "refId": "A"
        },
        {
          "expr": "rate(vllm:num_preemptions_total{job=~\"$job\"}[5m])",
          "legendFormat": "{{job}} preemptions/s",
          "refId": "B"
        }
      ]
    },
    {
      "id": 5,
      "type": "timeseries",
      "title": "Decode and prefill throughput",
      "description": "llama.cpp: llamacpp:predicted_tokens_seconds is decode, llamacpp:prompt_tokens_seconds is prompt. vLLM: the token counters, rate'd. The series carry the job label of their scrape, so the endpoint picker selects between the runtimes.",
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 16
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "targets": [
        {
          "expr": "llamacpp:predicted_tokens_seconds{job=~\"$job\"}",
          "legendFormat": "{{job}} decode tok/s (llama.cpp)",
          "refId": "A"
        },
        {
          "expr": "llamacpp:prompt_tokens_seconds{job=~\"$job\"}",
          "legendFormat": "{{job}} prefill tok/s (llama.cpp)",
          "refId": "B"
        },
        {
          "expr": "rate(vllm:generation_tokens_total{job=~\"$job\"}[5m])",
          "legendFormat": "{{job}} decode tok/s (vLLM)",
          "refId": "C"
        },
        {
          "expr": "rate(vllm:prompt_tokens_total{job=~\"$job\"}[5m])",
          "legendFormat": "{{job}} prefill tok/s (vLLM)",
          "refId": "D"
        }
      ]
    },
    {
      "id": 6,
      "type": "timeseries",
      "title": "Decode steps and batch size",
      "description": "llama.cpp-only signal \u2014 vLLM does not publish per-step counters. Steps per second and the busy-slot count per step (the effective batch).",
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 16
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "targets": [
        {
          "expr": "rate(llamacpp:n_decode_total{job=~\"$job\"}[5m])",
          "legendFormat": "{{job}} decode steps/s",
          "refId": "A"
        },
        {
          "expr": "llamacpp:n_busy_slots_per_decode{job=~\"$job\"}",
          "legendFormat": "{{job}} sequences per decode step",
          "refId": "B"
        }
      ]
    },
    {
      "id": 7,
      "type": "timeseries",
      "title": "Firing fleet alerts",
      "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 24
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "targets": [
        {
          "expr": "ALERTS{alertname=~\"Llm.*\", alertstate=\"firing\"}",
          "legendFormat": "{{alertname}} {{job}}",
          "refId": "A"
        }
      ]
    },
    {
      "id": 8,
      "type": "timeseries",
      "title": "Request latency p95 (vLLM)",
      "description": "vLLM-only \u2014 the histogram the p95 panel was retired for on 2026-09-16 (llama.cpp exports no latency histogram) is back with the radiance cutover. TTFT is what a fresh prefill costs; ITL is the per-token cadence an agent loop feels; e2e is the whole request.",
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 24
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "fieldConfig": {
        "defaults": {
          "unit": "s",
          "custom": {
            "fillOpacity": 10,
            "lineWidth": 1
          }
        },
        "overrides": []
      },
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:time_to_first_token_seconds_bucket{job=~\"$job\"}[5m])))",
          "legendFormat": "{{job}} TTFT p95",
          "refId": "A"
        },
        {
          "expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:inter_token_latency_seconds_bucket{job=~\"$job\"}[5m])))",
          "legendFormat": "{{job}} ITL p95",
          "refId": "B"
        },
        {
          "expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket{job=~\"$job\"}[5m])))",
          "legendFormat": "{{job}} e2e p95",
          "refId": "C"
        }
      ]
    }
  ]
}