dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

retire frank-medium: tests, planning and observability ground on frank-primary

Browse Source 
The frank-medium retirement tail the 2026-09-26 lucebox cutover left
uncommitted: the radiance quadlet tree, its swap/fetch tooling and its
seam test are deleted (archive copies live in dankserver/.AGENTTRASH/
20260926-frank-medium-radiance/ and scripts/.AGENTTRASH/20260926-
frank-medium-radiance/), the planner/allowlist/test-changed mappings
drop the service, and frank-large's chat template moves to scripts/
(its old home was the retired frank-medium dir).

Finished here:
- tests/frank_primary_unit.bats: the lucebox quadlet seam (boot
  persistence, host-network bind, HIP_VISIBLE_DEVICES=0,1, HIP-reserved
  env-name guard, checkout-mounted entrypoint/healthcheck, first-start
  patience, deployer payload/port gates).
- test-changed.sh: explicit .AGENTTRASH/ case — retirement commits move
  files to trash, and the wrapper refused those additions with "no
  focused-test mapping".
- observability: the vllm: namespace left with the radiance runtime.
  Rules keep only llamacpp:* series (prefix-cache reuse, spec-decode,
  queueing, endpoint-down); the dashboard drops the vLLM KV/latency
  panels; prometheus.yml documents that frank-primary (lucebox) has no
  /metrics and therefore no scrape job. llm-fleet.json was left
  corrupted mid-edit (missing opening brace) and is rewritten from HEAD.

Suite: 339 green (was 7 red).
This commit is contained in:
Frank committed 2026-09-26 19:25:46 -07:00
1 parent 32d264ccda
commit d7bf9f119a
22 files changed
+243 -2001
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

shared/grafana/provisioning/dashboards/llm-fleet.json
+7 -104
View File
@@ -1,15 +1,14 @@
{
"uid": "llm-fleet",
"title": "LLM fleet \u2014 capacity and reuse",
"description": "The signals that explain the LLM endpoints under load. The fleet runs two runtimes and the panels read both: vllm:* for frank-medium (Dankserver, vllm-radiance TP=2 across both R9700s since 2026-09-20) and llamacpp:* for frank-small (Dankclaw, RTX 3090) and frank-large (Dankclaw, gfx1151 iGPU \u2014 the CIRU llama.cpp build does export metrics, verified 2026-09-20). The endpoint picker selects between them. See serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md for the numbers these panels are read against, and serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split.",
"description": "The signals that explain the LLM endpoints under load. One metric namespace: llamacpp:* for frank-small (Dankclaw, RTX 3090) and frank-large (Dankclaw, gfx1151 iGPU). frank-primary (Dankserver, the Lucebox DS4.1 engine since the 2026-09-26 cutover) exports no metrics and has no panels here. The endpoint picker selects between them. See serverAppConfigs-docs/Infrastructure/Observability.md for the namespace split and the addressability rules.",
"tags": [
"llm",
"frank-medium",
"capacity"
],
"timezone": "browser",
"schemaVersion": 39,
"version": 1,
"version": 2,
"refresh": "30s",
"time": {
"from": "now-6h",
@@ -26,7 +25,7 @@
"uid": "prometheus"
},
"query": {
"query": "label_values({__name__=~\"vllm:kv_cache_usage_perc|llamacpp:.*\"}, job)",
"query": "label_values({__name__=~\"llamacpp:.*\"}, job)",
"refId": "A"
},
"includeAll": true,
@@ -43,7 +42,7 @@
"id": 1,
"type": "timeseries",
"title": "Prefix cache reuse rate",
"description": "rate(hits)/rate(queries) in tokens. On the llama.cpp side the prompt counter EXCLUDES cached tokens (its own HELP line), so that side divides by cached + prompt. High means pinned conversations are served from cache: measured 2026-09-20, an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused. Reuse on frank-medium happens at 1,648-token block granularity (the attention block size is padded to the mamba page size), so shared prefixes shorter than two blocks re-prefill.",
"description": "rate(hits)/rate(queries) in tokens. The llama.cpp prompt counter EXCLUDES cached tokens (its own HELP line), so this divides by cached + prompt. High means pinned conversations are served from cache: measured 2026-09-20, an identical 13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %) reused.",
"gridPos": {
"h": 8,
"w": 12,
@@ -71,11 +70,6 @@
"expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) + rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 0.001)",
"legendFormat": "{{job}} reuse (llama.cpp)",
"refId": "A"
},
{
"expr": "job:llm_vllm_prefix_cache_reuse:rate5m{job=~\"$job\"}",
"legendFormat": "{{job}} reuse (vLLM)",
"refId": "B"
}
]
},
@@ -83,7 +77,7 @@
"id": 2,
"type": "timeseries",
"title": "Speculative decoding (acceptance)",
"description": "The one failure no client can see: the drafter silently stops and the endpoint keeps answering at autoregressive speed. llama.cpp: accepted tokens per draft step. vLLM: mean accepted length (1 + accepted/drafts) from the recording rule job:llm_spec_accept_length:rate5m \u2014 measured 3.0-4.3 at boot 2026-09-20 with the DFlash2 drafter.",
"description": "The one failure no client can see: the drafter silently stops and the endpoint keeps answering at autoregressive speed. Accepted tokens per draft step, and the mean accepted length (1 + accepted/drafts, the recording rule job:llm_spec_accept_length:rate5m) \u2014 1.0 means the draft round is pure overhead.",
"gridPos": {
"h": 8,
"w": 12,
@@ -137,50 +131,12 @@
"expr": "llamacpp:requests_processing{job=~\"$job\"}",
"legendFormat": "{{job}} processing",
"refId": "B"
},
{
"expr": "vllm:num_requests_waiting{job=~\"$job\"}",
"legendFormat": "{{job}} waiting (vLLM)",
"refId": "C"
},
{
"expr": "vllm:num_requests_running{job=~\"$job\"}",
"legendFormat": "{{job}} running (vLLM)",
"refId": "D"
}
]
},
{
"id": 4,
"type": "timeseries",
"title": "KV pool fill (vLLM)",
"description": "The gauge that was retired 2026-09-16 for want of a llama.cpp equivalent and came back when frank-medium moved to vLLM: vllm:kv_cache_usage_perc (the pool is auto-sized \u2014 943,581 tokens at boot 2026-09-20). Sustained high fill with a collapsing reuse panel is the thrash; preemptions are the visible symptom of it. The llama.cpp side has no fill gauge: its capacity wall is panel 3.",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"targets": [
{
"expr": "vllm:kv_cache_usage_perc{job=~\"$job\"}",
"legendFormat": "{{job}} KV fill %",
"refId": "A"
},
{
"expr": "rate(vllm:num_preemptions_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} preemptions/s",
"refId": "B"
}
]
},
{
"id": 5,
"type": "timeseries",
"title": "Decode and prefill throughput",
"description": "llama.cpp: llamacpp:predicted_tokens_seconds is decode, llamacpp:prompt_tokens_seconds is prompt. vLLM: the token counters, rate'd. The series carry the job label of their scrape, so the endpoint picker selects between the runtimes.",
"gridPos": {
@@ -203,21 +159,11 @@
"expr": "llamacpp:prompt_tokens_seconds{job=~\"$job\"}",
"legendFormat": "{{job}} prefill tok/s (llama.cpp)",
"refId": "B"
},
{
"expr": "rate(vllm:generation_tokens_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} decode tok/s (vLLM)",
"refId": "C"
},
{
"expr": "rate(vllm:prompt_tokens_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} prefill tok/s (vLLM)",
"refId": "D"
}
]
},
{
"id": 6,
"id": 5,
"type": "timeseries",
"title": "Decode steps and batch size",
"description": "llama.cpp-only signal \u2014 vLLM does not publish per-step counters. Steps per second and the busy-slot count per step (the effective batch).",
@@ -245,7 +191,7 @@
]
},
{
"id": 7,
"id": 6,
"type": "timeseries",
"title": "Firing fleet alerts",
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
@@ -266,49 +212,6 @@
"refId": "A"
}
]
},
{
"id": 8,
"type": "timeseries",
"title": "Request latency p95 (vLLM)",
"description": "vLLM-only \u2014 the histogram the p95 panel was retired for on 2026-09-16 (llama.cpp exports no latency histogram) is back with the radiance cutover. TTFT is what a fresh prefill costs; ITL is the per-token cadence an agent loop feels; e2e is the whole request.",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 24
},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"fillOpacity": 10,
"lineWidth": 1
}
},
"overrides": []
},
"targets": [
{
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:time_to_first_token_seconds_bucket{job=~\"$job\"}[5m])))",
"legendFormat": "{{job}} TTFT p95",
"refId": "A"
},
{
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:inter_token_latency_seconds_bucket{job=~\"$job\"}[5m])))",
"legendFormat": "{{job}} ITL p95",
"refId": "B"
},
{
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket{job=~\"$job\"}[5m])))",
"legendFormat": "{{job}} e2e p95",
"refId": "C"
}
]
}
]
}
shared/prometheus/prometheus.yml
+9 -21
View File
@@ -79,19 +79,11 @@ scrape_configs:
# this container, not the guarded vhosts: these are machine clients and
# Prometheus cannot solve an Anubis challenge.
#
# Two metric namespaces, because the fleet runs two runtimes:
#
# * vllm:* — frank-medium (vllm-radiance TP=2 since 2026-09-20): prompt and
# generation token counters, num_requests_waiting plus its reason label,
# num_preemptions_total, the kv_cache_usage_perc fill gauge, the
# prefix_cache hits/queries pair, the spec_decode drafts/accepted triple,
# and real latency histograms (time_to_first_token, inter_token,
# e2e_request_latency).
# * llamacpp:* — frank-large and frank-small. Both take --metrics and both
# were verified 2026-09-20 to export requests_deferred (the queue depth,
# the only form llama.cpp shows the capacity wall in),
# spec_decode_num_drafts_total, tokens_predicted_total, and the
# prompt_tokens_cached_total / prompt_tokens_total reuse pair.
# One metric namespace: llamacpp:* — frank-large and frank-small, both
# llama.cpp with --metrics and both verified 2026-09-20 to export
# requests_deferred (the queue depth, the only form llama.cpp shows the
# capacity wall in), spec_decode_num_drafts_total, tokens_predicted_total,
# and the prompt_tokens_cached_total / prompt_tokens_total reuse pair.
#
# frank-large is scraped over host.containers.internal, not its own LAN address:
# this Prometheus is a bridge-networked container (Network=dankclaw_default),
@@ -102,14 +94,10 @@ scrape_configs:
# "dial tcp 192.168.0.143:7771") even though the endpoint was healthy and
# exporting: the target address was unreachable by construction, not the
# service down. Same host, same rule, so both local endpoints use the gateway.
# frank-medium stays a LAN address — it is a different machine.
- job_name: "frank-medium"
metrics_path: "/metrics"
static_configs:
- targets: ["192.168.0.235:7770"]
labels:
service: frank-medium
# (2026-09-26) frank-medium is replaced by frank-primary (lucebox DS4.1),
# which exposes no metrics endpoint (/metrics -> 404), so there is no job
# for it here. The endpoint's liveness is covered by the homepage probe;
# the documented remedy is a blackbox exporter, which is not deployed.
- job_name: "frank-small"
metrics_path: "/metrics"
static_configs:
shared/prometheus/rules/llm-fleet.yml
+43 -98
View File
@@ -1,32 +1,27 @@
# Alerting and recording rules for the LLM fleet.
#
# Two metric namespaces, because the fleet runs two runtimes:
# One metric namespace: llamacpp:* — frank-small and frank-large, both
# llama.cpp with --metrics (the family verified from /metrics on 2026-09-16
# and 2026-09-20). frank-primary (Dankserver, the Lucebox DS4.1 engine since
# the 2026-09-26 cutover) answers 404 on /metrics, so nothing here selects
# it; its liveness is covered by the homepage probe, and the documented
# remedy is a blackbox exporter (not deployed).
#
# * llamacpp:* — frank-small (Dankclaw, llama.cpp with --metrics).
# * vllm:* — frank-medium (Dankserver, vllm-radiance TP=2 since 2026-09-20).
# Every expression is keyed to a series that EXISTS. Nothing matches on
# absence: when an endpoint's runtime leaves the fleet, its selectors are
# deleted rather than left vacuous (the vllm: group went with frank-medium's
# radiance runtime on 2026-09-26).
#
# Every expression is keyed to a series that EXISTS, verified against
# 192.168.0.235:7770/metrics and 192.168.0.143:7771/metrics on 2026-09-20.
# Nothing here matches on absence: when frank-medium moved back to vLLM the
# llamacpp: selectors narrowed to frank-small rather than going vacuous, and
# the vllm: group below was added beside them, not in place of them.
# What llamacpp:* can and cannot see:
#
# What each namespace can and cannot see:
#
# * KV pressure. vLLM exports a fill gauge (vllm:kv_cache_usage_perc) and a
# preemption counter; llama.cpp exports neither, so its capacity wall is
# * KV pressure: llama.cpp exports no fill gauge, so the capacity wall is
# observed as queueing (llamacpp:requests_deferred).
# * Prefix reuse. Both export a hits/queries pair counted in tokens, but the
# units are NOT interchangeable: llama.cpp's prompt_tokens_total EXCLUDES
# cached tokens (its own HELP line says so), so that side must divide by
# cached + prompt to get a fraction, while vLLM divides hits by queries.
# The two recording rules below keep that straight, and the collapse alert
# watches both sides with one threshold.
# * Speculative decoding. Both export drafts and accepted tokens, so
# LlmSpecDecodeOff covers both runtimes.
# * Latency histograms exist only on vLLM (e2e, TTFT, inter-token). No
# p95 alert is defined for the llama.cpp side because there is nothing to
# compute one from.
# * Prefix reuse: counted in tokens, and prompt_tokens_total EXCLUDES
# cached tokens (its own HELP line says so), so a reuse fraction divides
# cached by cached + prompt.
# * Speculative decoding: drafts and accepted tokens are both counters,
# so LlmSpecDecodeOff watches the pair.
# * No latency histograms exist, so there is no p95 alert.
#
# IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
# Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
@@ -37,32 +32,24 @@ groups:
- name: llm-fleet
interval: 30s
rules:
# Recording rules so the dashboard and the collapse alert below share one
# expression. Both namespaces export hits/queries pairs in TOKENS; the
# rate of the quotient is the reuse rate. Kept as two records because the
# series names differ and a vacuous joint query is worse than two honest
# ones.
# Recording rules so the dashboard and the collapse alert below share
# one expression. llamacpp's prompt counter EXCLUDES cached tokens, so
# the reuse fraction is cached / (cached + prompt).
- record: job:llm_prefix_cache_reuse:rate5m
expr: |
rate(llamacpp:prompt_tokens_cached_total[5m])
/ clamp_min(rate(llamacpp:prompt_tokens_cached_total[5m])
+ rate(llamacpp:prompt_tokens_total[5m]), 0.001)
- record: job:llm_vllm_prefix_cache_reuse:rate5m
expr: |
rate(vllm:prefix_cache_hits_total[5m])
/ clamp_min(rate(vllm:prefix_cache_queries_total[5m]), 0.001)
# Mean accepted draft length (1.0 = the drafter proposes and nothing is
# taken, so the draft round is pure overhead). Measured 3.0-4.3 on
# frank-medium at boot with the DFlash2 drafter, 2026-09-20.
# taken, so the draft round is pure overhead).
- record: job:llm_spec_accept_length:rate5m
expr: |
1 + rate(vllm:spec_decode_num_accepted_tokens_total[5m])
/ clamp_min(rate(vllm:spec_decode_num_drafts_total[5m]), 0.001)
1 + rate(llamacpp:spec_decode_num_accepted_tokens_total[5m])
/ clamp_min(rate(llamacpp:spec_decode_num_drafts_total[5m]), 0.001)
# The capacity wall on the llama.cpp side, in the only form it exports:
# work waiting for a slot. Past here the next large request queues.
# The capacity wall, in the only form llama.cpp exports: work waiting
# for a slot. Past here the next large request queues.
- alert: LlmRequestsQueueing
expr: llamacpp:requests_deferred > 0
for: 10m
@@ -75,51 +62,15 @@ groups:
fleet is genuinely busy or one endpoint is carrying it alone. With
--parallel 1 a second concurrent request always shows up here.
# The capacity wall on the vLLM side, in its own two terms: requests the
# scheduler has not admitted, and requests it admitted and then PREEMPTED
# (a preemption throws away the KV and re-prefills, so its rate is the
# cost of running the pool full). frank-medium auto-sizes its pool to
# 943,581 tokens = 3.59 concurrent full 262,144-token conversations, so
# saturation is four such conversations, not two.
- alert: LlmRequestsWaiting
expr: vllm:num_requests_waiting > 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: vLLM has not admitted work for 10 minutes"
description: >-
The scheduler is holding requests back. Check the reason label on
vllm:num_requests_waiting_by_reason (capacity = no KV room, defer =
waiting for a better batch shape) and the preemption counter
vllm:num_preemptions_total.
- alert: LlmKvCacheSaturated
expr: vllm:kv_cache_usage_perc > 0.95
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: KV pool above 95 % for 10 minutes"
description: >-
The pool is full, which is where preemption and re-prefill start:
decode slows before anything errors. The pool is auto-sized by the
quadlet (no KV_CACHE_MEMORY_MB pin); the boot ledger prints the
token count — `podman logs frank-medium | grep "GPU KV cache size"`
— and it was 943,581 tokens on 2026-09-20.
# The thrash signature measured on 2026-09-16: requests keep arriving but
# almost none are served from cache, which is what several concurrent
# 50-80k-token conversations do to this fleet. One threshold, two
# namespaces, and a traffic floor per side so an idle endpoint cannot
# fire it on a 0/0.
# 50-80k-token conversations do to this fleet. One threshold with a
# traffic floor so an idle endpoint cannot fire it on a 0/0.
- alert: LlmPrefixCacheCollapse
expr: >-
(job:llm_prefix_cache_reuse:rate5m < 0.20
job:llm_prefix_cache_reuse:rate5m < 0.20
and rate(llamacpp:prompt_tokens_total[5m])
+ rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05)
or (job:llm_vllm_prefix_cache_reuse:rate5m < 0.20
and rate(vllm:prefix_cache_queries_total[5m]) > 1)
+ rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05
for: 15m
labels:
severity: info
@@ -132,14 +83,11 @@ groups:
# The speculative path is the one failure no client can see: the server
# answers normally, just at half the speed, when the drafter is gone.
# Drafts and accepts are counters in both runtimes, so a rate that stays
# at zero while the endpoint is serving tokens means nothing is drafted.
# Drafts and accepts are counters, so a rate that stays at zero while
# the endpoint is serving tokens means nothing is drafted.
- alert: LlmSpecDecodeOff
expr: >-
(rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1)
or (rate(vllm:spec_decode_num_drafts_total[5m]) == 0
and rate(vllm:generation_tokens_total[5m]) > 0.1)
expr: rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1
for: 15m
labels:
severity: warning
@@ -147,11 +95,9 @@ groups:
summary: "{{ $labels.job }}: serving without speculative decoding"
description: >-
Tokens are being produced but no draft has been proposed for 15
minutes. frank-small (llama.cpp) should log "adding speculative
implementation" at load; frank-medium (vllm-radiance) should be
counting vllm:spec_decode_num_drafts_total and reporting 3+ mean
accept length (job:llm_spec_accept_length:rate5m). Either way the
endpoint is decoding autoregressively.
minutes. The llama.cpp load log should carry an "adding
speculative implementation" line; the endpoint is decoding
autoregressively.
# The endpoint, not a replica: one process serves each port now, so a
# restart of the unit is a scrape gap, not a failover.
@@ -163,10 +109,9 @@ groups:
annotations:
summary: "{{ $labels.job }}: scrape target down for 2 minutes"
description: >-
Prometheus cannot reach the LLM endpoint. frank-medium is a
rootless quadlet: `systemctl --user is-active frank-medium` on
Dankserver, then `podman ps -a --filter name=frank-medium`.
frank-small is a root unit: `systemctl is-active frank-small` on
Dankclaw. All three endpoints are scraped: frank-large's CIRU
llama.cpp build does export the llamacpp:* family (verified
2026-09-20), so this alert covers the whole fleet.
Prometheus cannot reach the LLM endpoint. frank-small and
frank-large are root units on Dankclaw: `systemctl is-active
frank-small` / `frank-large`. Both scraped endpoints export the
llamacpp:* family. frank-primary (Dankserver) is not scraped —
its runtime exports no metrics; check the homepage probe or
`systemctl --user is-active frank-primary` on Dankserver.