dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

retire frank-medium: tests, planning and observability ground on frank-primary

Browse Source 
The frank-medium retirement tail the 2026-09-26 lucebox cutover left
uncommitted: the radiance quadlet tree, its swap/fetch tooling and its
seam test are deleted (archive copies live in dankserver/.AGENTTRASH/
20260926-frank-medium-radiance/ and scripts/.AGENTTRASH/20260926-
frank-medium-radiance/), the planner/allowlist/test-changed mappings
drop the service, and frank-large's chat template moves to scripts/
(its old home was the retired frank-medium dir).

Finished here:
- tests/frank_primary_unit.bats: the lucebox quadlet seam (boot
  persistence, host-network bind, HIP_VISIBLE_DEVICES=0,1, HIP-reserved
  env-name guard, checkout-mounted entrypoint/healthcheck, first-start
  patience, deployer payload/port gates).
- test-changed.sh: explicit .AGENTTRASH/ case — retirement commits move
  files to trash, and the wrapper refused those additions with "no
  focused-test mapping".
- observability: the vllm: namespace left with the radiance runtime.
  Rules keep only llamacpp:* series (prefix-cache reuse, spec-decode,
  queueing, endpoint-down); the dashboard drops the vLLM KV/latency
  panels; prometheus.yml documents that frank-primary (lucebox) has no
  /metrics and therefore no scrape job. llm-fleet.json was left
  corrupted mid-edit (missing opening brace) and is rewritten from HEAD.

Suite: 339 green (was 7 red).
This commit is contained in:
Frank
2026-09-26 19:25:46 -07:00
parent 32d264ccda
commit d7bf9f119a
22 changed files with 243 additions and 2001 deletions
Download Patch File Download Diff File Expand all files Collapse all files
shared/prometheus/rules/llm-fleet.yml
+43 -98
View File
@@ -1,32 +1,27 @@
# Alerting and recording rules for the LLM fleet.
#
# Two metric namespaces, because the fleet runs two runtimes:
# One metric namespace: llamacpp:* — frank-small and frank-large, both
# llama.cpp with --metrics (the family verified from /metrics on 2026-09-16
# and 2026-09-20). frank-primary (Dankserver, the Lucebox DS4.1 engine since
# the 2026-09-26 cutover) answers 404 on /metrics, so nothing here selects
# it; its liveness is covered by the homepage probe, and the documented
# remedy is a blackbox exporter (not deployed).
#
# * llamacpp:* — frank-small (Dankclaw, llama.cpp with --metrics).
# * vllm:* — frank-medium (Dankserver, vllm-radiance TP=2 since 2026-09-20).
# Every expression is keyed to a series that EXISTS. Nothing matches on
# absence: when an endpoint's runtime leaves the fleet, its selectors are
# deleted rather than left vacuous (the vllm: group went with frank-medium's
# radiance runtime on 2026-09-26).
#
# Every expression is keyed to a series that EXISTS, verified against
# 192.168.0.235:7770/metrics and 192.168.0.143:7771/metrics on 2026-09-20.
# Nothing here matches on absence: when frank-medium moved back to vLLM the
# llamacpp: selectors narrowed to frank-small rather than going vacuous, and
# the vllm: group below was added beside them, not in place of them.
# What llamacpp:* can and cannot see:
#
# What each namespace can and cannot see:
#
# * KV pressure. vLLM exports a fill gauge (vllm:kv_cache_usage_perc) and a
# preemption counter; llama.cpp exports neither, so its capacity wall is
# * KV pressure: llama.cpp exports no fill gauge, so the capacity wall is
# observed as queueing (llamacpp:requests_deferred).
# * Prefix reuse. Both export a hits/queries pair counted in tokens, but the
# units are NOT interchangeable: llama.cpp's prompt_tokens_total EXCLUDES
# cached tokens (its own HELP line says so), so that side must divide by
# cached + prompt to get a fraction, while vLLM divides hits by queries.
# The two recording rules below keep that straight, and the collapse alert
# watches both sides with one threshold.
# * Speculative decoding. Both export drafts and accepted tokens, so
# LlmSpecDecodeOff covers both runtimes.
# * Latency histograms exist only on vLLM (e2e, TTFT, inter-token). No
# p95 alert is defined for the llama.cpp side because there is nothing to
# compute one from.
# * Prefix reuse: counted in tokens, and prompt_tokens_total EXCLUDES
# cached tokens (its own HELP line says so), so a reuse fraction divides
# cached by cached + prompt.
# * Speculative decoding: drafts and accepted tokens are both counters,
# so LlmSpecDecodeOff watches the pair.
# * No latency histograms exist, so there is no p95 alert.
#
# IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
# Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
@@ -37,32 +32,24 @@ groups:
- name: llm-fleet
interval: 30s
rules:
# Recording rules so the dashboard and the collapse alert below share one
# expression. Both namespaces export hits/queries pairs in TOKENS; the
# rate of the quotient is the reuse rate. Kept as two records because the
# series names differ and a vacuous joint query is worse than two honest
# ones.
# Recording rules so the dashboard and the collapse alert below share
# one expression. llamacpp's prompt counter EXCLUDES cached tokens, so
# the reuse fraction is cached / (cached + prompt).
- record: job:llm_prefix_cache_reuse:rate5m
expr: |
rate(llamacpp:prompt_tokens_cached_total[5m])
/ clamp_min(rate(llamacpp:prompt_tokens_cached_total[5m])
+ rate(llamacpp:prompt_tokens_total[5m]), 0.001)
- record: job:llm_vllm_prefix_cache_reuse:rate5m
expr: |
rate(vllm:prefix_cache_hits_total[5m])
/ clamp_min(rate(vllm:prefix_cache_queries_total[5m]), 0.001)
# Mean accepted draft length (1.0 = the drafter proposes and nothing is
# taken, so the draft round is pure overhead). Measured 3.0-4.3 on
# frank-medium at boot with the DFlash2 drafter, 2026-09-20.
# taken, so the draft round is pure overhead).
- record: job:llm_spec_accept_length:rate5m
expr: |
1 + rate(vllm:spec_decode_num_accepted_tokens_total[5m])
/ clamp_min(rate(vllm:spec_decode_num_drafts_total[5m]), 0.001)
1 + rate(llamacpp:spec_decode_num_accepted_tokens_total[5m])
/ clamp_min(rate(llamacpp:spec_decode_num_drafts_total[5m]), 0.001)
# The capacity wall on the llama.cpp side, in the only form it exports:
# work waiting for a slot. Past here the next large request queues.
# The capacity wall, in the only form llama.cpp exports: work waiting
# for a slot. Past here the next large request queues.
- alert: LlmRequestsQueueing
expr: llamacpp:requests_deferred > 0
for: 10m
@@ -75,51 +62,15 @@ groups:
fleet is genuinely busy or one endpoint is carrying it alone. With
--parallel 1 a second concurrent request always shows up here.
# The capacity wall on the vLLM side, in its own two terms: requests the
# scheduler has not admitted, and requests it admitted and then PREEMPTED
# (a preemption throws away the KV and re-prefills, so its rate is the
# cost of running the pool full). frank-medium auto-sizes its pool to
# 943,581 tokens = 3.59 concurrent full 262,144-token conversations, so
# saturation is four such conversations, not two.
- alert: LlmRequestsWaiting
expr: vllm:num_requests_waiting > 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: vLLM has not admitted work for 10 minutes"
description: >-
The scheduler is holding requests back. Check the reason label on
vllm:num_requests_waiting_by_reason (capacity = no KV room, defer =
waiting for a better batch shape) and the preemption counter
vllm:num_preemptions_total.
- alert: LlmKvCacheSaturated
expr: vllm:kv_cache_usage_perc > 0.95
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: KV pool above 95 % for 10 minutes"
description: >-
The pool is full, which is where preemption and re-prefill start:
decode slows before anything errors. The pool is auto-sized by the
quadlet (no KV_CACHE_MEMORY_MB pin); the boot ledger prints the
token count — `podman logs frank-medium | grep "GPU KV cache size"`
— and it was 943,581 tokens on 2026-09-20.
# The thrash signature measured on 2026-09-16: requests keep arriving but
# almost none are served from cache, which is what several concurrent
# 50-80k-token conversations do to this fleet. One threshold, two
# namespaces, and a traffic floor per side so an idle endpoint cannot
# fire it on a 0/0.
# 50-80k-token conversations do to this fleet. One threshold with a
# traffic floor so an idle endpoint cannot fire it on a 0/0.
- alert: LlmPrefixCacheCollapse
expr: >-
(job:llm_prefix_cache_reuse:rate5m < 0.20
job:llm_prefix_cache_reuse:rate5m < 0.20
and rate(llamacpp:prompt_tokens_total[5m])
+ rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05)
or (job:llm_vllm_prefix_cache_reuse:rate5m < 0.20
and rate(vllm:prefix_cache_queries_total[5m]) > 1)
+ rate(llamacpp:prompt_tokens_cached_total[5m]) > 0.05
for: 15m
labels:
severity: info
@@ -132,14 +83,11 @@ groups:
# The speculative path is the one failure no client can see: the server
# answers normally, just at half the speed, when the drafter is gone.
# Drafts and accepts are counters in both runtimes, so a rate that stays
# at zero while the endpoint is serving tokens means nothing is drafted.
# Drafts and accepts are counters, so a rate that stays at zero while
# the endpoint is serving tokens means nothing is drafted.
- alert: LlmSpecDecodeOff
expr: >-
(rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1)
or (rate(vllm:spec_decode_num_drafts_total[5m]) == 0
and rate(vllm:generation_tokens_total[5m]) > 0.1)
expr: rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1
for: 15m
labels:
severity: warning
@@ -147,11 +95,9 @@ groups:
summary: "{{ $labels.job }}: serving without speculative decoding"
description: >-
Tokens are being produced but no draft has been proposed for 15
minutes. frank-small (llama.cpp) should log "adding speculative
implementation" at load; frank-medium (vllm-radiance) should be
counting vllm:spec_decode_num_drafts_total and reporting 3+ mean
accept length (job:llm_spec_accept_length:rate5m). Either way the
endpoint is decoding autoregressively.
minutes. The llama.cpp load log should carry an "adding
speculative implementation" line; the endpoint is decoding
autoregressively.
# The endpoint, not a replica: one process serves each port now, so a
# restart of the unit is a scrape gap, not a failover.
@@ -163,10 +109,9 @@ groups:
annotations:
summary: "{{ $labels.job }}: scrape target down for 2 minutes"
description: >-
Prometheus cannot reach the LLM endpoint. frank-medium is a
rootless quadlet: `systemctl --user is-active frank-medium` on
Dankserver, then `podman ps -a --filter name=frank-medium`.
frank-small is a root unit: `systemctl is-active frank-small` on
Dankclaw. All three endpoints are scraped: frank-large's CIRU
llama.cpp build does export the llamacpp:* family (verified
2026-09-20), so this alert covers the whole fleet.
Prometheus cannot reach the LLM endpoint. frank-small and
frank-large are root units on Dankclaw: `systemctl is-active
frank-small` / `frank-large`. Both scraped endpoints export the
llamacpp:* family. frank-primary (Dankserver) is not scraped —
its runtime exports no metrics; check the homepage probe or
`systemctl --user is-active frank-primary` on Dankserver.