feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets
The radiance TP=2 cutover changed the metric namespace frank-medium exports (llamacpp:* -> vllm:*) and left two scrape targets addressing host processes by an address a container cannot reach. - shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never had), the job:llm_vllm_prefix_cache_reuse:rate5m and job:llm_spec_accept_length:rate5m recording rules, and re-cut LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with per-family traffic floors so an idle endpoint cannot fire them. Every expression validated against live Prometheus. - shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771 and frank-large scraped at host.containers.internal:7770. Both are HOST processes on Dankclaw and netavark DNATs published container ports only, so the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp build does export the llamacpp:* family, contra the old comment. - tests/observability_quadlet.bats: the two seams now encode the new contract (per-endpoint reachability by process type; every named series must be on the measured list for its engine; engine-independent rules must name both). - homepage docs.html/services.json: frank-medium sees pictures, frank-large is CIRU Orca with a single seat (was halogen w4b, two seats, 200k). - docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified capabilities, rollback, replica-era sections labelled); llama.cpp runbook carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured verdict; Observability records the namespace split and the DNAT lesson; AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook index flipped; both indexes updated; work-log entry added. Also repaired agent-pass damage: prose prepended above frontmatter openers, duplicated runbook bodies, a literal tool-call token pasted into a runbook, and an index destroyed by diff artifacts (restored from HEAD, flips re-applied). Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
This commit is contained in:
Frank
@@ -47,10 +47,10 @@
|
||||
|
||||
<h2>frank-large — the big brain</h2>
|
||||
<ul>
|
||||
<li><strong>Model:</strong> Qwen3.8-Flash-Next, w4b quantization, served by halogen-flash-server.</li>
|
||||
<li><strong>Model:</strong> Qwen3.8-Flash CIRU-STRIX-Orca, served by CIRU v4.4.1 llama.cpp with the model's own MTP draft head.</li>
|
||||
<li><strong>Hardware:</strong> Dankclaw's Strix Halo iGPU (gfx1151), shared with the whole host.</li>
|
||||
<li><strong>Best for:</strong> long, thoughtful text conversations. 200k-token context per conversation.</li>
|
||||
<li><strong>Capacity:</strong> exactly two conversations resident at once. A third waits — politely, but it waits.</li>
|
||||
<li><strong>Best for:</strong> long, thoughtful text conversations at the model's full trained window.</li>
|
||||
<li><strong>Capacity:</strong> exactly one conversation resident at a time. A second waits — politely, but it waits.</li>
|
||||
<li><strong>Not for:</strong> images. It cannot see them.</li>
|
||||
</ul>
|
||||
|
||||
@@ -67,11 +67,11 @@
|
||||
<li><strong>Model:</strong> Qwen3.8-27B-Uncensored at MXFP4, on vllm-radiance with a DFlash2 draft head.</li>
|
||||
<li><strong>Hardware:</strong> Dankserver's two Radeon R9700s as ONE tensor-parallel engine (TP=2).</li>
|
||||
<li><strong>Best for:</strong> general chat and reasoning when the big brain is busy; the biggest pure-language model we run, and the fastest we have measured.</li>
|
||||
<li><strong>Capacity:</strong> 262k-token context, shared by up to eight conversations at once. Text only — pictures go to <code>frank-small</code>.</li>
|
||||
<li><strong>Capacity:</strong> 262k-token context shared by up to eight conversations at once. It can see pictures too; <code>frank-small</code> is the faster eye and holds four conversations.</li>
|
||||
</ul>
|
||||
|
||||
<p class="foot-note">Which one to pick? Screenshots and photos: <code>frank-small</code>.
|
||||
Deep long-text work: <code>frank-large</code> (mind the two-seat limit).
|
||||
Deep long-text work: <code>frank-large</code> (mind the single seat).
|
||||
When in doubt: <code>frank-medium</code>.</p>
|
||||
</section>
|
||||
|
||||
|
||||
@@ -233,7 +233,7 @@
|
||||
{
|
||||
"name": "frank-large",
|
||||
"category": "user",
|
||||
"description": "The big brain. Qwen3.8-Flash-Next w4b on halogen-flash-server, running on Dankclaw's Strix Halo iGPU. Text-only; holds two 200k-token conversations resident.",
|
||||
"description": "The big brain. Qwen3.8-Flash CIRU-STRIX-Orca on CIRU v4.4.1 llama.cpp, running on Dankclaw's Strix Halo iGPU. Text-only; one conversation resident at a time.",
|
||||
"url": "",
|
||||
"docs": "/docs#models",
|
||||
"probe": "127.0.0.1:7770",
|
||||
|
||||
@@ -69,34 +69,51 @@ scrape_configs:
|
||||
- "192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
|
||||
- "192.168.0.143:3005" # OpenWebUI
|
||||
|
||||
# LLM fleet. Scraped on the LAN addresses, not the guarded vhosts: these are
|
||||
# machine clients and Prometheus cannot solve an Anubis challenge.
|
||||
# Both LLM endpoints are llama.cpp, so both export the same llamacpp:
|
||||
# counter family: prompt/decode throughput, requests_deferred (queue depth
|
||||
# — the only form llama.cpp exposes the capacity wall in), the
|
||||
# spec_decode_num_drafts / spec_decode_num_accepted pair (proof the drafter
|
||||
# is running), and the prefix-cache pair prompt_tokens_cached_total over
|
||||
# prompt_tokens_total (their rates are the reuse rate, which says whether
|
||||
# pinned conversations are actually being reused).
|
||||
# frank-large is deliberately NOT scraped: the halogen engine that serves it
|
||||
# publishes /health, /cache and /v1/* and no Prometheus text endpoint, so a
|
||||
# target here would be red by construction and would fire LlmEndpointDown
|
||||
# forever. Its liveness is on the homepage (a loopback probe) and in the
|
||||
# engine's own boot ledger. The remedy, when the fleet wants it, is the
|
||||
# blackbox exporter already wanted for the comfyui/services probes.
|
||||
# LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
|
||||
# this container, not the guarded vhosts: these are machine clients and
|
||||
# Prometheus cannot solve an Anubis challenge.
|
||||
#
|
||||
# frank-small is llama.cpp (its serve script passes --metrics) and does
|
||||
# export text-format counters under llamacpp: on the port its unit publishes.
|
||||
# Two metric namespaces, because the fleet runs two runtimes:
|
||||
#
|
||||
# * vllm:* — frank-medium (vllm-radiance TP=2 since 2026-09-20): prompt and
|
||||
# generation token counters, num_requests_waiting plus its reason label,
|
||||
# num_preemptions_total, the kv_cache_usage_perc fill gauge, the
|
||||
# prefix_cache hits/queries pair, the spec_decode drafts/accepted triple,
|
||||
# and real latency histograms (time_to_first_token, inter_token,
|
||||
# e2e_request_latency).
|
||||
# * llamacpp:* — frank-large and frank-small. Both take --metrics and both
|
||||
# were verified 2026-09-20 to export requests_deferred (the queue depth,
|
||||
# the only form llama.cpp shows the capacity wall in),
|
||||
# spec_decode_num_drafts_total, tokens_predicted_total, and the
|
||||
# prompt_tokens_cached_total / prompt_tokens_total reuse pair.
|
||||
#
|
||||
# frank-large is scraped over host.containers.internal, not its own LAN address:
|
||||
# this Prometheus is a bridge-networked container (Network=dankclaw_default),
|
||||
# and on this netavark host a container can only reach the host's LAN IP at a
|
||||
# PUBLISHED port — a host-process bind like llama-server's 0.0.0.0:7770 is
|
||||
# refused at 192.168.0.143:7770 while it answers at 169.254.1.2:7770. That is
|
||||
# why frank-small was red here for days (up{job="frank-small"} == 0,
|
||||
# "dial tcp 192.168.0.143:7771") even though the endpoint was healthy and
|
||||
# exporting: the target address was unreachable by construction, not the
|
||||
# service down. Same host, same rule, so both local endpoints use the gateway.
|
||||
# frank-medium stays a LAN address — it is a different machine.
|
||||
- job_name: "frank-medium"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.235:7770"]
|
||||
labels:
|
||||
service: frank-medium
|
||||
|
||||
|
||||
- job_name: "frank-small"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.143:7771"]
|
||||
- targets: ["host.containers.internal:7771"]
|
||||
labels:
|
||||
service: frank-small
|
||||
|
||||
- job_name: "frank-large"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
- targets: ["host.containers.internal:7770"]
|
||||
labels:
|
||||
service: frank-large
|
||||
|
||||
@@ -1,39 +1,64 @@
|
||||
# Alerting and recording rules for the LLM fleet.
|
||||
#
|
||||
# Every expression here is keyed to a series that EXISTS as of 2026-09-16. When
|
||||
# frank-medium moved from vllm-radiance to llama.cpp the exported namespace
|
||||
# changed with it (vllm: -> llamacpp:), and two signals the old file alerted on
|
||||
# no longer exist in any form:
|
||||
# Two metric namespaces, because the fleet runs two runtimes:
|
||||
#
|
||||
# * vllm:kv_cache_usage_perc — llama.cpp exports no KV-fill gauge. Its
|
||||
# observable consequence is queueing, which is what LlmRequestsQueueing
|
||||
# watches instead. A renamed query would have been a rule that can never
|
||||
# fire.
|
||||
# * vllm:e2e_request_latency_seconds_* — no latency histogram either. The
|
||||
# rule it fed (p95 wall clock) is retired rather than restated, and
|
||||
# LlmSpecDecodeOff takes its place: the failure an operator actually
|
||||
# cannot see from the outside on this deployment.
|
||||
# * llamacpp:* — frank-small (Dankclaw, llama.cpp with --metrics).
|
||||
# * vllm:* — frank-medium (Dankserver, vllm-radiance TP=2 since 2026-09-20).
|
||||
#
|
||||
# Every expression is keyed to a series that EXISTS, verified against
|
||||
# 192.168.0.235:7770/metrics and 192.168.0.143:7771/metrics on 2026-09-20.
|
||||
# Nothing here matches on absence: when frank-medium moved back to vLLM the
|
||||
# llamacpp: selectors narrowed to frank-small rather than going vacuous, and
|
||||
# the vllm: group below was added beside them, not in place of them.
|
||||
#
|
||||
# What each namespace can and cannot see:
|
||||
#
|
||||
# * KV pressure. vLLM exports a fill gauge (vllm:kv_cache_usage_perc) and a
|
||||
# preemption counter; llama.cpp exports neither, so its capacity wall is
|
||||
# observed as queueing (llamacpp:requests_deferred).
|
||||
# * Prefix reuse. Both export a hits/queries pair, in tokens; the two
|
||||
# recording rules below keep the units straight and the collapse alert
|
||||
# watches both sides with one threshold.
|
||||
# * Speculative decoding. Both export drafts and accepted tokens, so
|
||||
# LlmSpecDecodeOff covers both runtimes.
|
||||
# * Latency histograms exist only on vLLM (e2e, TTFT, inter-token). No
|
||||
# p95 alert is defined for the llama.cpp side because there is nothing to
|
||||
# compute one from.
|
||||
#
|
||||
# IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
|
||||
# Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
|
||||
# alertmanager.container), whose single receiver webhooks the host-side
|
||||
# signal-relay user unit, which hands the text to signal-cli under the
|
||||
# existing account on this host. Before that date a firing alert appeared on
|
||||
# Prometheus' /alerts page, in the ALERTS metric (which the LLM fleet
|
||||
# dashboard graphs) and nowhere else.
|
||||
# existing account on this host.
|
||||
groups:
|
||||
- name: llm-fleet
|
||||
interval: 30s
|
||||
rules:
|
||||
# Recording rule so the dashboard and the alert below share one
|
||||
# expression. The exported series are counters; the rate is the reuse.
|
||||
# Recording rules so the dashboard and the collapse alert below share one
|
||||
# expression. Both namespaces export hits/queries pairs in TOKENS; the
|
||||
# rate of the quotient is the reuse rate. Kept as two records because the
|
||||
# series names differ and a vacuous joint query is worse than two honest
|
||||
# ones.
|
||||
- record: job:llm_prefix_cache_reuse:rate5m
|
||||
expr: |
|
||||
rate(llamacpp:prompt_tokens_cached_total[5m])
|
||||
/ clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001)
|
||||
|
||||
# The capacity wall, in the only form llama.cpp exports it: work waiting
|
||||
# for the single slot. Past here the next large request queues.
|
||||
- record: job:llm_vllm_prefix_cache_reuse:rate5m
|
||||
expr: |
|
||||
rate(vllm:prefix_cache_hits_total[5m])
|
||||
/ clamp_min(rate(vllm:prefix_cache_queries_total[5m]), 0.001)
|
||||
|
||||
# Mean accepted draft length (1.0 = the drafter proposes and nothing is
|
||||
# taken, so the draft round is pure overhead). Measured 3.0-4.3 on
|
||||
# frank-medium at boot with the DFlash2 drafter, 2026-09-20.
|
||||
- record: job:llm_spec_accept_length:rate5m
|
||||
expr: |
|
||||
1 + rate(vllm:spec_decode_num_accepted_tokens_total[5m])
|
||||
/ clamp_min(rate(vllm:spec_decode_num_drafts_total[5m]), 0.001)
|
||||
|
||||
# The capacity wall on the llama.cpp side, in the only form it exports:
|
||||
# work waiting for a slot. Past here the next large request queues.
|
||||
- alert: LlmRequestsQueueing
|
||||
expr: llamacpp:requests_deferred > 0
|
||||
for: 10m
|
||||
@@ -46,13 +71,50 @@ groups:
|
||||
fleet is genuinely busy or one endpoint is carrying it alone. With
|
||||
--parallel 1 a second concurrent request always shows up here.
|
||||
|
||||
# The capacity wall on the vLLM side, in its own two terms: requests the
|
||||
# scheduler has not admitted, and requests it admitted and then PREEMPTED
|
||||
# (a preemption throws away the KV and re-prefills, so its rate is the
|
||||
# cost of running the pool full). frank-medium auto-sizes its pool to
|
||||
# 943,581 tokens = 3.59 concurrent full 262,144-token conversations, so
|
||||
# saturation is four such conversations, not two.
|
||||
- alert: LlmRequestsWaiting
|
||||
expr: vllm:num_requests_waiting > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: vLLM has not admitted work for 10 minutes"
|
||||
description: >-
|
||||
The scheduler is holding requests back. Check the reason label on
|
||||
vllm:num_requests_waiting_by_reason (capacity = no KV room, defer =
|
||||
waiting for a better batch shape) and the preemption counter
|
||||
vllm:num_preemptions_total.
|
||||
|
||||
- alert: LlmKvCacheSaturated
|
||||
expr: vllm:kv_cache_usage_perc > 0.95
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: KV pool above 95 % for 10 minutes"
|
||||
description: >-
|
||||
The pool is full, which is where preemption and re-prefill start:
|
||||
decode slows before anything errors. The pool is auto-sized by the
|
||||
quadlet (no KV_CACHE_MEMORY_MB pin); the boot ledger prints the
|
||||
token count — `podman logs frank-medium | grep "GPU KV cache size"`
|
||||
— and it was 943,581 tokens on 2026-09-20.
|
||||
|
||||
# The thrash signature measured on 2026-09-16: requests keep arriving but
|
||||
# almost none are served from cache, which is what several concurrent
|
||||
# 50-80k-token conversations do to this fleet.
|
||||
# 50-80k-token conversations do to this fleet. One threshold, two
|
||||
# namespaces, and a traffic floor per side so an idle endpoint cannot
|
||||
# fire it on a 0/0.
|
||||
- alert: LlmPrefixCacheCollapse
|
||||
expr: >-
|
||||
job:llm_prefix_cache_reuse:rate5m < 0.20
|
||||
and rate(llamacpp:prompt_tokens_total[5m]) > 0.05
|
||||
(job:llm_prefix_cache_reuse:rate5m < 0.20
|
||||
and rate(llamacpp:prompt_tokens_total[5m]) > 0.05)
|
||||
or (job:llm_vllm_prefix_cache_reuse:rate5m < 0.20
|
||||
and rate(vllm:prefix_cache_queries_total[5m]) > 1)
|
||||
for: 15m
|
||||
labels:
|
||||
severity: info
|
||||
@@ -65,12 +127,14 @@ groups:
|
||||
|
||||
# The speculative path is the one failure no client can see: the server
|
||||
# answers normally, just at half the speed, when the drafter is gone.
|
||||
# Drafts and accepts are counters, so a rate that stays at zero while the
|
||||
# endpoint is serving tokens means nothing is being drafted.
|
||||
# Drafts and accepts are counters in both runtimes, so a rate that stays
|
||||
# at zero while the endpoint is serving tokens means nothing is drafted.
|
||||
- alert: LlmSpecDecodeOff
|
||||
expr: >-
|
||||
rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
|
||||
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1
|
||||
(rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
|
||||
and rate(llamacpp:tokens_predicted_total[5m]) > 0.1)
|
||||
or (rate(vllm:spec_decode_num_drafts_total[5m]) == 0
|
||||
and rate(vllm:generation_tokens_total[5m]) > 0.1)
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
@@ -78,11 +142,11 @@ groups:
|
||||
summary: "{{ $labels.job }}: serving without speculative decoding"
|
||||
description: >-
|
||||
Tokens are being produced but no draft has been proposed for 15
|
||||
minutes. On the llama.cpp endpoints that means the drafter is not
|
||||
running (check the unit's log for the
|
||||
"adding speculative implementation" line — draft-mtp on both
|
||||
endpoints as of 2026-09-17), and the endpoint is decoding
|
||||
autoregressively.
|
||||
minutes. frank-small (llama.cpp) should log "adding speculative
|
||||
implementation" at load; frank-medium (vllm-radiance) should be
|
||||
counting vllm:spec_decode_num_drafts_total and reporting 3+ mean
|
||||
accept length (job:llm_spec_accept_length:rate5m). Either way the
|
||||
endpoint is decoding autoregressively.
|
||||
|
||||
# The endpoint, not a replica: one process serves each port now, so a
|
||||
# restart of the unit is a scrape gap, not a failover.
|
||||
@@ -94,8 +158,10 @@ groups:
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: scrape target down for 2 minutes"
|
||||
description: >-
|
||||
Prometheus cannot reach the LLM endpoint. Check the units:
|
||||
`systemctl is-active frank-medium` on Dankserver (a root llama.cpp
|
||||
unit) and `systemctl is-active frank-small` on Dankclaw. frank-large
|
||||
is not scraped at all — the halogen engine exports no metrics — so
|
||||
this alert can only ever name the endpoints that do export one.
|
||||
Prometheus cannot reach the LLM endpoint. frank-medium is a
|
||||
rootless quadlet: `systemctl --user is-active frank-medium` on
|
||||
Dankserver, then `podman ps -a --filter name=frank-medium`.
|
||||
frank-small is a root unit: `systemctl is-active frank-small` on
|
||||
Dankclaw. frank-large is not scraped at all — its CIRU runtime
|
||||
serves no Prometheus text endpoint — so this alert can only ever
|
||||
name the endpoints that do export one.
|
||||
|
||||
Reference in New Issue
Block a user