frank-medium: serve the whole 262144-token window, and repoint every signal to llama.cpp
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and 29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a 140,069-token prompt prefilled through the single slot in 134 s to prove the window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this runtime's. Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to 140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance, and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt. The signals move with the runtime, so the fleet's rules and dashboard are re-cut rather than renamed: vllm: series do not exist any more, and two of the old rules watched things llama.cpp does not export at all (KV-pool fill, p95 latency). Those are retired with the reason written down, and LlmSpecDecodeOff takes one of the slots -- a lost drafter is the failure no client can see. The observability seam now fails if any rule names a series the endpoint does not export, which is what would have caught the stale vllm: expressions. omp models.yml (outside the repo) is updated in the same pass: contextWindow 262144, and the full reasoning-effort map, because Qwen3.8's template answers 500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer 200. Subagents run at high.
This commit is contained in:
Frank
@@ -1,7 +1,18 @@
|
||||
# Alerting and recording rules for the LLM fleet.
|
||||
#
|
||||
# Thresholds are not guesses: each one corresponds to a failure mode measured on
|
||||
# this fleet on 2026-09-16 (see serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md).
|
||||
# Every expression here is keyed to a series that EXISTS as of 2026-09-16. When
|
||||
# frank-medium moved from vllm-radiance to llama.cpp the exported namespace
|
||||
# changed with it (vllm: -> llamacpp:), and two signals the old file alerted on
|
||||
# no longer exist in any form:
|
||||
#
|
||||
# * vllm:kv_cache_usage_perc — llama.cpp exports no KV-fill gauge. Its
|
||||
# observable consequence is queueing, which is what LlmRequestsQueueing
|
||||
# watches instead. A renamed query would have been a rule that can never
|
||||
# fire.
|
||||
# * vllm:e2e_request_latency_seconds_* — no latency histogram either. The
|
||||
# rule it fed (p95 wall clock) is retired rather than restated, and
|
||||
# LlmSpecDecodeOff takes its place: the failure an operator actually
|
||||
# cannot see from the outside on this deployment.
|
||||
#
|
||||
# IMPORTANT: nothing is notified. There is no Alertmanager in this stack, so a
|
||||
# firing alert appears on Prometheus' /alerts page, in the ALERTS metric (which
|
||||
@@ -17,42 +28,30 @@ groups:
|
||||
# expression. The exported series are counters; the rate is the reuse.
|
||||
- record: job:llm_prefix_cache_reuse:rate5m
|
||||
expr: |
|
||||
rate(vllm:prefix_cache_hits_total[5m])
|
||||
/ clamp_min(rate(vllm:prefix_cache_queries_total[5m]), 0.001)
|
||||
rate(llamacpp:prompt_tokens_cached_total[5m])
|
||||
/ clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001)
|
||||
|
||||
# The capacity wall. Each R9700 carries 122,636 KV tokens; past ~90 %
|
||||
# the next large request queues or evicts a cached prefix.
|
||||
- alert: LlmKvPoolSaturated
|
||||
expr: vllm:kv_cache_usage_perc > 0.90
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: KV pool above 90 % for 10 minutes"
|
||||
description: >-
|
||||
A card's KV pool is nearly full. Expect queueing or cold re-prefills
|
||||
for large requests; the wall and its arithmetic are in the
|
||||
frank-medium runbook.
|
||||
|
||||
# Concurrency limit reached: MAXSEQS=4 per replica.
|
||||
# The capacity wall, in the only form llama.cpp exports it: work waiting
|
||||
# for the single slot. Past here the next large request queues.
|
||||
- alert: LlmRequestsQueueing
|
||||
expr: vllm:num_requests_waiting > 0
|
||||
expr: llamacpp:requests_deferred > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: requests waiting for 10 minutes"
|
||||
description: >-
|
||||
All sequence slots are busy and work is queueing. Either the fleet is
|
||||
genuinely busy or one endpoint is carrying it alone.
|
||||
The endpoint's slot(s) are busy and work is queueing. Either the
|
||||
fleet is genuinely busy or one endpoint is carrying it alone. With
|
||||
--parallel 1 a second concurrent request always shows up here.
|
||||
|
||||
# The thrash signature measured on 2026-09-16: requests keep arriving but
|
||||
# almost none are served from cache, which is what three concurrent
|
||||
# 50-80k-token conversations do to a two-card fleet.
|
||||
# almost none are served from cache, which is what several concurrent
|
||||
# 50-80k-token conversations do to this fleet.
|
||||
- alert: LlmPrefixCacheCollapse
|
||||
expr: >-
|
||||
job:llm_prefix_cache_reuse:rate5m < 0.20
|
||||
and rate(vllm:prefix_cache_queries_total[5m]) > 0.05
|
||||
and rate(llamacpp:prompt_tokens_total[5m]) > 0.05
|
||||
for: 15m
|
||||
labels:
|
||||
severity: info
|
||||
@@ -61,12 +60,29 @@ groups:
|
||||
description: >-
|
||||
Nearly every request is prefilling from scratch. If several large
|
||||
contexts are in flight together, this is the capacity wall rather
|
||||
than a routing fault — placement and admission were both measured
|
||||
and neither fixes it.
|
||||
than a routing fault.
|
||||
|
||||
# The endpoint, not the replica: two replicas serve :7770 through the
|
||||
# proxy, so this fires only when both are down (a single restart is
|
||||
# invisible — the proxy fails over).
|
||||
# The speculative path is the one failure no client can see: the server
|
||||
# answers normally, just at half the speed, when the drafter is gone.
|
||||
# Drafts and accepts are counters, so a rate that stays at zero while the
|
||||
# endpoint is serving tokens means nothing is being drafted.
|
||||
- alert: LlmSpecDecodeOff
|
||||
expr: >-
|
||||
rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
|
||||
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: serving without speculative decoding"
|
||||
description: >-
|
||||
Tokens are being produced but no draft has been proposed for 15
|
||||
minutes. On frank-medium that means the DFlash2 drafter is not
|
||||
running (check the unit's log for common_speculative_impl_draft_dflash),
|
||||
and the endpoint is decoding autoregressively.
|
||||
|
||||
# The endpoint, not a replica: one process serves each port now, so a
|
||||
# restart of the unit is a scrape gap, not a failover.
|
||||
- alert: LlmEndpointDown
|
||||
expr: up{job=~"frank-.*"} == 0
|
||||
for: 2m
|
||||
@@ -76,24 +92,7 @@ groups:
|
||||
summary: "{{ $labels.job }}: scrape target down for 2 minutes"
|
||||
description: >-
|
||||
Prometheus cannot reach the LLM endpoint. Check the units:
|
||||
`systemctl --user is-active frank-medium-{a,b,proxy}` on Dankserver,
|
||||
`systemctl is-active frank-small` (the llama.cpp root unit) or
|
||||
`systemctl --user is-active frank-large` (the halogen quadlet) on
|
||||
Dankclaw. frank-large is not scraped at all — the halogen engine
|
||||
exports no metrics — so this alert can only ever name the endpoints
|
||||
that do export one.
|
||||
|
||||
# What an operator actually feels: wall-clock per request, prefill
|
||||
# included. A cold 50k-token prefill alone is ~22 s on one card.
|
||||
- alert: LlmLatencyHigh
|
||||
expr: >-
|
||||
histogram_quantile(0.95,
|
||||
sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket[10m]))) > 60
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: p95 request latency above 60 s for 10 minutes"
|
||||
description: >-
|
||||
Requests are taking a minute or more end to end. On this hardware
|
||||
that is usually large contexts competing for one card's KV pool.
|
||||
`systemctl is-active frank-medium` on Dankserver (a root llama.cpp
|
||||
unit) and `systemctl is-active frank-small` on Dankclaw. frank-large
|
||||
is not scraped at all — the halogen engine exports no metrics — so
|
||||
this alert can only ever name the endpoints that do export one.
|
||||
|
||||
Reference in New Issue
Block a user