dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets

Browse Source 
The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.

- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
  LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
  had), the job:llm_vllm_prefix_cache_reuse:rate5m and
  job:llm_spec_accept_length:rate5m recording rules, and re-cut
  LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
  per-family traffic floors so an idle endpoint cannot fire them. Every
  expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
  and frank-large scraped at host.containers.internal:7770. Both are HOST
  processes on Dankclaw and netavark DNATs published container ports only, so
  the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
  build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
  (per-endpoint reachability by process type; every named series must be on the
  measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
  CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
  capabilities, rollback, replica-era sections labelled); llama.cpp runbook
  carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
  verdict; Observability records the namespace split and the DNAT lesson;
  AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
  index flipped; both indexes updated; work-log entry added.

Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
This commit is contained in:
Frank
2026-09-20 19:35:56 -07:00
parent c2116658f8
commit 08f1c7ad09
13 changed files with 687 additions and 286 deletions
Download Patch File Download Diff File Expand all files Collapse all files
shared/prometheus/rules/llm-fleet.yml
+102 -36
View File
@@ -1,39 +1,64 @@
# Alerting and recording rules for the LLM fleet.
#
# Every expression here is keyed to a series that EXISTS as of 2026-09-16. When
# frank-medium moved from vllm-radiance to llama.cpp the exported namespace
# changed with it (vllm: -> llamacpp:), and two signals the old file alerted on
# no longer exist in any form:
# Two metric namespaces, because the fleet runs two runtimes:
#
# * vllm:kv_cache_usage_perc — llama.cpp exports no KV-fill gauge. Its
# observable consequence is queueing, which is what LlmRequestsQueueing
# watches instead. A renamed query would have been a rule that can never
# fire.
# * vllm:e2e_request_latency_seconds_* — no latency histogram either. The
# rule it fed (p95 wall clock) is retired rather than restated, and
# LlmSpecDecodeOff takes its place: the failure an operator actually
# cannot see from the outside on this deployment.
# * llamacpp:* — frank-small (Dankclaw, llama.cpp with --metrics).
# * vllm:* — frank-medium (Dankserver, vllm-radiance TP=2 since 2026-09-20).
#
# Every expression is keyed to a series that EXISTS, verified against
# 192.168.0.235:7770/metrics and 192.168.0.143:7771/metrics on 2026-09-20.
# Nothing here matches on absence: when frank-medium moved back to vLLM the
# llamacpp: selectors narrowed to frank-small rather than going vacuous, and
# the vllm: group below was added beside them, not in place of them.
#
# What each namespace can and cannot see:
#
# * KV pressure. vLLM exports a fill gauge (vllm:kv_cache_usage_perc) and a
# preemption counter; llama.cpp exports neither, so its capacity wall is
# observed as queueing (llamacpp:requests_deferred).
# * Prefix reuse. Both export a hits/queries pair, in tokens; the two
# recording rules below keep the units straight and the collapse alert
# watches both sides with one threshold.
# * Speculative decoding. Both export drafts and accepted tokens, so
# LlmSpecDecodeOff covers both runtimes.
# * Latency histograms exist only on vLLM (e2e, TTFT, inter-token). No
# p95 alert is defined for the llama.cpp side because there is nothing to
# compute one from.
#
# IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
# Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
# alertmanager.container), whose single receiver webhooks the host-side
# signal-relay user unit, which hands the text to signal-cli under the
# existing account on this host. Before that date a firing alert appeared on
# Prometheus' /alerts page, in the ALERTS metric (which the LLM fleet
# dashboard graphs) and nowhere else.
# existing account on this host.
groups:
- name: llm-fleet
interval: 30s
rules:
# Recording rule so the dashboard and the alert below share one
# expression. The exported series are counters; the rate is the reuse.
# Recording rules so the dashboard and the collapse alert below share one
# expression. Both namespaces export hits/queries pairs in TOKENS; the
# rate of the quotient is the reuse rate. Kept as two records because the
# series names differ and a vacuous joint query is worse than two honest
# ones.
- record: job:llm_prefix_cache_reuse:rate5m
expr: |
rate(llamacpp:prompt_tokens_cached_total[5m])
/ clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001)
# The capacity wall, in the only form llama.cpp exports it: work waiting
# for the single slot. Past here the next large request queues.
- record: job:llm_vllm_prefix_cache_reuse:rate5m
expr: |
rate(vllm:prefix_cache_hits_total[5m])
/ clamp_min(rate(vllm:prefix_cache_queries_total[5m]), 0.001)
# Mean accepted draft length (1.0 = the drafter proposes and nothing is
# taken, so the draft round is pure overhead). Measured 3.0-4.3 on
# frank-medium at boot with the DFlash2 drafter, 2026-09-20.
- record: job:llm_spec_accept_length:rate5m
expr: |
1 + rate(vllm:spec_decode_num_accepted_tokens_total[5m])
/ clamp_min(rate(vllm:spec_decode_num_drafts_total[5m]), 0.001)
# The capacity wall on the llama.cpp side, in the only form it exports:
# work waiting for a slot. Past here the next large request queues.
- alert: LlmRequestsQueueing
expr: llamacpp:requests_deferred > 0
for: 10m
@@ -46,13 +71,50 @@ groups:
fleet is genuinely busy or one endpoint is carrying it alone. With
--parallel 1 a second concurrent request always shows up here.
# The capacity wall on the vLLM side, in its own two terms: requests the
# scheduler has not admitted, and requests it admitted and then PREEMPTED
# (a preemption throws away the KV and re-prefills, so its rate is the
# cost of running the pool full). frank-medium auto-sizes its pool to
# 943,581 tokens = 3.59 concurrent full 262,144-token conversations, so
# saturation is four such conversations, not two.
- alert: LlmRequestsWaiting
expr: vllm:num_requests_waiting > 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: vLLM has not admitted work for 10 minutes"
description: >-
The scheduler is holding requests back. Check the reason label on
vllm:num_requests_waiting_by_reason (capacity = no KV room, defer =
waiting for a better batch shape) and the preemption counter
vllm:num_preemptions_total.
- alert: LlmKvCacheSaturated
expr: vllm:kv_cache_usage_perc > 0.95
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: KV pool above 95 % for 10 minutes"
description: >-
The pool is full, which is where preemption and re-prefill start:
decode slows before anything errors. The pool is auto-sized by the
quadlet (no KV_CACHE_MEMORY_MB pin); the boot ledger prints the
token count — `podman logs frank-medium | grep "GPU KV cache size"`
— and it was 943,581 tokens on 2026-09-20.
# The thrash signature measured on 2026-09-16: requests keep arriving but
# almost none are served from cache, which is what several concurrent
# 50-80k-token conversations do to this fleet.
# 50-80k-token conversations do to this fleet. One threshold, two
# namespaces, and a traffic floor per side so an idle endpoint cannot
# fire it on a 0/0.
- alert: LlmPrefixCacheCollapse
expr: >-
job:llm_prefix_cache_reuse:rate5m < 0.20
and rate(llamacpp:prompt_tokens_total[5m]) > 0.05
(job:llm_prefix_cache_reuse:rate5m < 0.20
and rate(llamacpp:prompt_tokens_total[5m]) > 0.05)
or (job:llm_vllm_prefix_cache_reuse:rate5m < 0.20
and rate(vllm:prefix_cache_queries_total[5m]) > 1)
for: 15m
labels:
severity: info
@@ -65,12 +127,14 @@ groups:
# The speculative path is the one failure no client can see: the server
# answers normally, just at half the speed, when the drafter is gone.
# Drafts and accepts are counters, so a rate that stays at zero while the
# endpoint is serving tokens means nothing is being drafted.
# Drafts and accepts are counters in both runtimes, so a rate that stays
# at zero while the endpoint is serving tokens means nothing is drafted.
- alert: LlmSpecDecodeOff
expr: >-
rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1
(rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1)
or (rate(vllm:spec_decode_num_drafts_total[5m]) == 0
and rate(vllm:generation_tokens_total[5m]) > 0.1)
for: 15m
labels:
severity: warning
@@ -78,11 +142,11 @@ groups:
summary: "{{ $labels.job }}: serving without speculative decoding"
description: >-
Tokens are being produced but no draft has been proposed for 15
minutes. On the llama.cpp endpoints that means the drafter is not
running (check the unit's log for the
"adding speculative implementation" line — draft-mtp on both
endpoints as of 2026-09-17), and the endpoint is decoding
autoregressively.
minutes. frank-small (llama.cpp) should log "adding speculative
implementation" at load; frank-medium (vllm-radiance) should be
counting vllm:spec_decode_num_drafts_total and reporting 3+ mean
accept length (job:llm_spec_accept_length:rate5m). Either way the
endpoint is decoding autoregressively.
# The endpoint, not a replica: one process serves each port now, so a
# restart of the unit is a scrape gap, not a failover.
@@ -94,8 +158,10 @@ groups:
annotations:
summary: "{{ $labels.job }}: scrape target down for 2 minutes"
description: >-
Prometheus cannot reach the LLM endpoint. Check the units:
`systemctl is-active frank-medium` on Dankserver (a root llama.cpp
unit) and `systemctl is-active frank-small` on Dankclaw. frank-large
is not scraped at all — the halogen engine exports no metrics — so
this alert can only ever name the endpoints that do export one.
Prometheus cannot reach the LLM endpoint. frank-medium is a
rootless quadlet: `systemctl --user is-active frank-medium` on
Dankserver, then `podman ps -a --filter name=frank-medium`.
frank-small is a root unit: `systemctl is-active frank-small` on
Dankclaw. frank-large is not scraped at all — its CIRU runtime
serves no Prometheus text endpoint — so this alert can only ever
name the endpoints that do export one.