dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: serve the whole 262144-token window, and repoint every signal to llama.cpp

Browse Source 
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.

Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.

The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.

omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.
This commit is contained in:
Frank
2026-09-16 15:29:40 -07:00
parent ab51d61198
commit dc76dc5dc4
8 changed files with 220 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
shared/grafana/provisioning/dashboards/llm-fleet.json
+25 -25
View File
@@ -1,7 +1,7 @@
{
"uid": "llm-fleet",
"title": "LLM fleet — capacity and reuse",
"description": "The signals that explain frank-medium under load, from the 2026-09-16 capacity work. The panels read the endpoint's own counters: KV fill, queue depth, prefix-cache reuse, and request latency. frank-small (llama.cpp on Dankclaw's RTX 3090) exports throughput counters and appears in the same endpoint picker; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md for the numbers these panels are read against.",
"title": "LLM fleet \u2014 capacity and reuse",
"description": "The signals that explain the LLM endpoints under load, from the 2026-09-16 capacity work. The panels read llama.cpp's own counters: speculative acceptance, queue depth, prefix-cache reuse, decode step rate and throughput. frank-medium (Dankserver, both R9700s, layer split) and frank-small (Dankclaw, RTX 3090) both export them; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md for the numbers these panels are read against.",
"tags": [
"llm",
"frank-medium",
@@ -42,7 +42,7 @@
{
"id": 1,
"type": "timeseries",
"title": "KV pool fill (vllm:kv_cache_usage_perc)",
"title": "Speculative decoding (DFlash2 acceptance)",
"description": "How full each card's KV pool is. One card holds ~122,636 tokens; sustained high values with a cold prefix cache is the thrash this fleet measured.",
"gridPos": {
"h": 8,
@@ -69,8 +69,13 @@
"targets": [
{
"refId": "A",
"expr": "vllm:kv_cache_usage_perc{job=~\"$job\"} * 100",
"legendFormat": "{{job}} engine {{engine}}"
"expr": "rate(llamacpp:spec_decode_num_accepted_tokens_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total{job=~\"$job\"}[5m]), 0.001)",
"legendFormat": "{{job}} accepted per draft step"
},
{
"refId": "B",
"expr": "rate(llamacpp:spec_decode_num_draft_tokens_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} draft tokens/s"
}
]
},
@@ -103,13 +108,13 @@
"targets": [
{
"refId": "A",
"expr": "vllm:num_requests_waiting{job=~\"$job\"}",
"legendFormat": "{{job}} waiting"
"expr": "llamacpp:requests_deferred{job=~\"$job\"}",
"legendFormat": "{{job}} deferred"
},
{
"refId": "B",
"expr": "vllm:num_requests_running{job=~\"$job\"}",
"legendFormat": "{{job}} running"
"expr": "llamacpp:requests_processing{job=~\"$job\"}",
"legendFormat": "{{job}} processing"
}
]
},
@@ -143,7 +148,7 @@
"targets": [
{
"refId": "A",
"expr": "rate(vllm:prefix_cache_hits_total{job=~\"$job\"}[5m]) / clamp_min(rate(vllm:prefix_cache_queries_total{job=~\"$job\"}[5m]), 1)",
"expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 1)",
"legendFormat": "{{job}} reuse"
}
]
@@ -151,8 +156,8 @@
{
"id": 4,
"type": "timeseries",
"title": "End-to-end request latency p95",
"description": "Wall-clock per request, prefill included — the number an agent-loop turn feels.",
"title": "Decode steps and batch size",
"description": "Wall-clock per request, prefill included \u2014 the number an agent-loop turn feels.",
"gridPos": {
"h": 8,
"w": 12,
@@ -176,8 +181,13 @@
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket{job=~\"$job\"}[5m])))",
"legendFormat": "{{job}} p95"
"expr": "rate(llamacpp:n_decode_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} decode steps/s"
},
{
"refId": "B",
"expr": "llamacpp:n_busy_slots_per_decode{job=~\"$job\"}",
"legendFormat": "{{job}} sequences per decode step"
}
]
},
@@ -207,16 +217,6 @@
"overrides": []
},
"targets": [
{
"refId": "A",
"expr": "rate(vllm:generation_tokens_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} generated tok/s"
},
{
"refId": "B",
"expr": "rate(vllm:prompt_tokens_total{job=~\"$job\"}[5m])",
"legendFormat": "{{job}} prompt tok/s"
},
{
"refId": "C",
"expr": "llamacpp:predicted_tokens_seconds{job=~\"$job\"}",
@@ -233,7 +233,7 @@
"id": 6,
"type": "timeseries",
"title": "Alerts firing (ALERTS)",
"description": "Prometheus rules for the LLM fleet (shared/prometheus/rules/llm-fleet.yml): pool saturation, queueing, prefix-cache collapse, endpoint down, p95 latency. A firing alert appears here, on Prometheus' /alerts page, and nowhere else — this stack has no Alertmanager, so nothing is notified.",
"description": "Prometheus rules for the LLM fleet (shared/prometheus/rules/llm-fleet.yml): pool saturation, queueing, prefix-cache collapse, endpoint down, p95 latency. A firing alert appears here, on Prometheus' /alerts page, and nowhere else \u2014 this stack has no Alertmanager, so nothing is notified.",
"gridPos": {
"h": 8,
"w": 12,
shared/prometheus/rules/llm-fleet.yml
+50 -51
View File
@@ -1,7 +1,18 @@
# Alerting and recording rules for the LLM fleet.
#
# Thresholds are not guesses: each one corresponds to a failure mode measured on
# this fleet on 2026-09-16 (see serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md).
# Every expression here is keyed to a series that EXISTS as of 2026-09-16. When
# frank-medium moved from vllm-radiance to llama.cpp the exported namespace
# changed with it (vllm: -> llamacpp:), and two signals the old file alerted on
# no longer exist in any form:
#
# * vllm:kv_cache_usage_perc — llama.cpp exports no KV-fill gauge. Its
# observable consequence is queueing, which is what LlmRequestsQueueing
# watches instead. A renamed query would have been a rule that can never
# fire.
# * vllm:e2e_request_latency_seconds_* — no latency histogram either. The
# rule it fed (p95 wall clock) is retired rather than restated, and
# LlmSpecDecodeOff takes its place: the failure an operator actually
# cannot see from the outside on this deployment.
#
# IMPORTANT: nothing is notified. There is no Alertmanager in this stack, so a
# firing alert appears on Prometheus' /alerts page, in the ALERTS metric (which
@@ -17,42 +28,30 @@ groups:
# expression. The exported series are counters; the rate is the reuse.
- record: job:llm_prefix_cache_reuse:rate5m
expr: |
rate(vllm:prefix_cache_hits_total[5m])
/ clamp_min(rate(vllm:prefix_cache_queries_total[5m]), 0.001)
rate(llamacpp:prompt_tokens_cached_total[5m])
/ clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001)
# The capacity wall. Each R9700 carries 122,636 KV tokens; past ~90 %
# the next large request queues or evicts a cached prefix.
- alert: LlmKvPoolSaturated
expr: vllm:kv_cache_usage_perc > 0.90
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: KV pool above 90 % for 10 minutes"
description: >-
A card's KV pool is nearly full. Expect queueing or cold re-prefills
for large requests; the wall and its arithmetic are in the
frank-medium runbook.
# Concurrency limit reached: MAXSEQS=4 per replica.
# The capacity wall, in the only form llama.cpp exports it: work waiting
# for the single slot. Past here the next large request queues.
- alert: LlmRequestsQueueing
expr: vllm:num_requests_waiting > 0
expr: llamacpp:requests_deferred > 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: requests waiting for 10 minutes"
description: >-
All sequence slots are busy and work is queueing. Either the fleet is
genuinely busy or one endpoint is carrying it alone.
The endpoint's slot(s) are busy and work is queueing. Either the
fleet is genuinely busy or one endpoint is carrying it alone. With
--parallel 1 a second concurrent request always shows up here.
# The thrash signature measured on 2026-09-16: requests keep arriving but
# almost none are served from cache, which is what three concurrent
# 50-80k-token conversations do to a two-card fleet.
# almost none are served from cache, which is what several concurrent
# 50-80k-token conversations do to this fleet.
- alert: LlmPrefixCacheCollapse
expr: >-
job:llm_prefix_cache_reuse:rate5m < 0.20
and rate(vllm:prefix_cache_queries_total[5m]) > 0.05
and rate(llamacpp:prompt_tokens_total[5m]) > 0.05
for: 15m
labels:
severity: info
@@ -61,12 +60,29 @@ groups:
description: >-
Nearly every request is prefilling from scratch. If several large
contexts are in flight together, this is the capacity wall rather
than a routing fault — placement and admission were both measured
and neither fixes it.
than a routing fault.
# The endpoint, not the replica: two replicas serve :7770 through the
# proxy, so this fires only when both are down (a single restart is
# invisible — the proxy fails over).
# The speculative path is the one failure no client can see: the server
# answers normally, just at half the speed, when the drafter is gone.
# Drafts and accepts are counters, so a rate that stays at zero while the
# endpoint is serving tokens means nothing is being drafted.
- alert: LlmSpecDecodeOff
expr: >-
rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: serving without speculative decoding"
description: >-
Tokens are being produced but no draft has been proposed for 15
minutes. On frank-medium that means the DFlash2 drafter is not
running (check the unit's log for common_speculative_impl_draft_dflash),
and the endpoint is decoding autoregressively.
# The endpoint, not a replica: one process serves each port now, so a
# restart of the unit is a scrape gap, not a failover.
- alert: LlmEndpointDown
expr: up{job=~"frank-.*"} == 0
for: 2m
@@ -76,24 +92,7 @@ groups:
summary: "{{ $labels.job }}: scrape target down for 2 minutes"
description: >-
Prometheus cannot reach the LLM endpoint. Check the units:
`systemctl --user is-active frank-medium-{a,b,proxy}` on Dankserver,
`systemctl is-active frank-small` (the llama.cpp root unit) or
`systemctl --user is-active frank-large` (the halogen quadlet) on
Dankclaw. frank-large is not scraped at all — the halogen engine
exports no metrics — so this alert can only ever name the endpoints
that do export one.
# What an operator actually feels: wall-clock per request, prefill
# included. A cold 50k-token prefill alone is ~22 s on one card.
- alert: LlmLatencyHigh
expr: >-
histogram_quantile(0.95,
sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket[10m]))) > 60
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: p95 request latency above 60 s for 10 minutes"
description: >-
Requests are taking a minute or more end to end. On this hardware
that is usually large contexts competing for one card's KV pool.
`systemctl is-active frank-medium` on Dankserver (a root llama.cpp
unit) and `systemctl is-active frank-small` on Dankclaw. frank-large
is not scraped at all — the halogen engine exports no metrics — so
this alert can only ever name the endpoints that do export one.