retire frank-medium: tests, planning and observability ground on frank-primary
The frank-medium retirement tail the 2026-09-26 lucebox cutover left uncommitted: the radiance quadlet tree, its swap/fetch tooling and its seam test are deleted (archive copies live in dankserver/.AGENTTRASH/ 20260926-frank-medium-radiance/ and scripts/.AGENTTRASH/20260926- frank-medium-radiance/), the planner/allowlist/test-changed mappings drop the service, and frank-large's chat template moves to scripts/ (its old home was the retired frank-medium dir). Finished here: - tests/frank_primary_unit.bats: the lucebox quadlet seam (boot persistence, host-network bind, HIP_VISIBLE_DEVICES=0,1, HIP-reserved env-name guard, checkout-mounted entrypoint/healthcheck, first-start patience, deployer payload/port gates). - test-changed.sh: explicit .AGENTTRASH/ case — retirement commits move files to trash, and the wrapper refused those additions with "no focused-test mapping". - observability: the vllm: namespace left with the radiance runtime. Rules keep only llamacpp:* series (prefix-cache reuse, spec-decode, queueing, endpoint-down); the dashboard drops the vLLM KV/latency panels; prometheus.yml documents that frank-primary (lucebox) has no /metrics and therefore no scrape job. llm-fleet.json was left corrupted mid-edit (missing opening brace) and is rewritten from HEAD. Suite: 339 green (was 7 red).
This commit is contained in:
Frank committed
1 parent
32d264ccda
commit
d7bf9f119a
22 files changed
+243
-2001
No files matched your search
@@ -1,15 +1,14 @@
|
||||
{
|
||||
"uid": "llm-fleet",
|
||||
"title": "LLM fleet \u2014 capacity and reuse",
|
||||
"description": "The signals that explain the LLM endpoints under load. The fleet runs two runtimes and the panels read both: vllm:* for frank-medium (Dankserver, vllm-radiance TP=2 across both R9700s since 2026-09-20) and llamacpp:* for frank-small (Dankclaw, RTX 3090) and frank-large (Dankclaw, gfx1151 iGPU \u2014 the CIRU llama.cpp build does export metrics, verified 2026-09-20). The endpoint picker selects between them. See serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md for the numbers these panels are read against, and serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split.",
|
||||
"description": "The signals that explain the LLM endpoints under load. One metric namespace: llamacpp:* for frank-small (Dankclaw, RTX 3090) and frank-large (Dankclaw, gfx1151 iGPU). frank-primary (Dankserver, the Lucebox DS4.1 engine since the 2026-09-26 cutover) exports no metrics and has no panels here. The endpoint picker selects between them. See serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split and the addressability rules.",
|
||||
"tags": [
|
||||
"llm",
|
||||
"frank-medium",
|
||||
"capacity"
|
||||
],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"version": 2,
|
||||
"refresh": "30s",
|
||||
"time": {
|
||||
"from": "now-6h",
|
||||
@@ -26,7 +25,7 @@
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"query": {
|
||||
"query": "label_values({__name__=~\"vllm:kv_cache_usage_perc|llamacpp:.*\"}, job)",
|
||||
"query": "label_values({__name__=~\"llamacpp:.*\"}, job)",
|
||||
"refId": "A"
|
||||
},
|
||||
"includeAll": true,
|
||||
@@ -43,7 +42,7 @@
|
||||
"id": 1,
|
||||
"type": "timeseries",
|
||||
"title": "Prefix cache reuse rate",
|
||||
"description": "rate(hits)/rate(queries) in tokens. On the llama.cpp side the prompt counter EXCLUDES cached tokens (its own HELP line), so that side divides by cached + prompt. High means pinned conversations are served from cache: measured 2026-09-20, an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused. Reuse on frank-medium happens at 1,648-token block granularity (the attention block size is padded to the mamba page size), so shared prefixes shorter than two blocks re-prefill.",
|
||||
"description": "rate(hits)/rate(queries) in tokens. The llama.cpp prompt counter EXCLUDES cached tokens (its own HELP line), so this divides by cached + prompt. High means pinned conversations are served from cache: measured 2026-09-20, an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
@@ -71,11 +70,6 @@
|
||||
"expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) + rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 0.001)",
|
||||
"legendFormat": "{{job}} reuse (llama.cpp)",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "job:llm_vllm_prefix_cache_reuse:rate5m{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} reuse (vLLM)",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -83,7 +77,7 @@
|
||||
"id": 2,
|
||||
"type": "timeseries",
|
||||
"title": "Speculative decoding (acceptance)",
|
||||
"description": "The one failure no client can see: the drafter silently stops and the endpoint keeps answering at autoregressive speed. llama.cpp: accepted tokens per draft step. vLLM: mean accepted length (1 + accepted/drafts) from the recording rule job:llm_spec_accept_length:rate5m \u2014 measured 3.0-4.3 at boot 2026-09-20 with the DFlash2 drafter.",
|
||||
"description": "The one failure no client can see: the drafter silently stops and the endpoint keeps answering at autoregressive speed. Accepted tokens per draft step, and the mean accepted length (1 + accepted/drafts, the recording rule job:llm_spec_accept_length:rate5m) \u2014 1.0 means the draft round is pure overhead.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
@@ -137,50 +131,12 @@
|
||||
"expr": "llamacpp:requests_processing{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} processing",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "vllm:num_requests_waiting{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} waiting (vLLM)",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"expr": "vllm:num_requests_running{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} running (vLLM)",
|
||||
"refId": "D"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "timeseries",
|
||||
"title": "KV pool fill (vLLM)",
|
||||
"description": "The gauge that was retired 2026-09-16 for want of a llama.cpp equivalent and came back when frank-medium moved to vLLM: vllm:kv_cache_usage_perc (the pool is auto-sized \u2014 943,581 tokens at boot 2026-09-20). Sustained high fill with a collapsing reuse panel is the thrash; preemptions are the visible symptom of it. The llama.cpp side has no fill gauge: its capacity wall is panel 3.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "vllm:kv_cache_usage_perc{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} KV fill %",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "rate(vllm:num_preemptions_total{job=~\"$job\"}[5m])",
|
||||
"legendFormat": "{{job}} preemptions/s",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Decode and prefill throughput",
|
||||
"description": "llama.cpp: llamacpp:predicted_tokens_seconds is decode, llamacpp:prompt_tokens_seconds is prompt. vLLM: the token counters, rate'd. The series carry the job label of their scrape, so the endpoint picker selects between the runtimes.",
|
||||
"gridPos": {
|
||||
@@ -203,21 +159,11 @@
|
||||
"expr": "llamacpp:prompt_tokens_seconds{job=~\"$job\"}",
|
||||
"legendFormat": "{{job}} prefill tok/s (llama.cpp)",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "rate(vllm:generation_tokens_total{job=~\"$job\"}[5m])",
|
||||
"legendFormat": "{{job}} decode tok/s (vLLM)",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"expr": "rate(vllm:prompt_tokens_total{job=~\"$job\"}[5m])",
|
||||
"legendFormat": "{{job}} prefill tok/s (vLLM)",
|
||||
"refId": "D"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Decode steps and batch size",
|
||||
"description": "llama.cpp-only signal \u2014 vLLM does not publish per-step counters. Steps per second and the busy-slot count per step (the effective batch).",
|
||||
@@ -245,7 +191,7 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Firing fleet alerts",
|
||||
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
|
||||
@@ -266,49 +212,6 @@
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "timeseries",
|
||||
"title": "Request latency p95 (vLLM)",
|
||||
"description": "vLLM-only \u2014 the histogram the p95 panel was retired for on 2026-09-16 (llama.cpp exports no latency histogram) is back with the radiance cutover. TTFT is what a fresh prefill costs; ITL is the per-token cadence an agent loop feels; e2e is the whole request.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 24
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:time_to_first_token_seconds_bucket{job=~\"$job\"}[5m])))",
|
||||
"legendFormat": "{{job}} TTFT p95",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:inter_token_latency_seconds_bucket{job=~\"$job\"}[5m])))",
|
||||
"legendFormat": "{{job}} ITL p95",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket{job=~\"$job\"}[5m])))",
|
||||
"legendFormat": "{{job}} e2e p95",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in new issue
Block a user