feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets
The radiance TP=2 cutover changed the metric namespace frank-medium exports (llamacpp:* -> vllm:*) and left two scrape targets addressing host processes by an address a container cannot reach. - shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never had), the job:llm_vllm_prefix_cache_reuse:rate5m and job:llm_spec_accept_length:rate5m recording rules, and re-cut LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with per-family traffic floors so an idle endpoint cannot fire them. Every expression validated against live Prometheus. - shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771 and frank-large scraped at host.containers.internal:7770. Both are HOST processes on Dankclaw and netavark DNATs published container ports only, so the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp build does export the llamacpp:* family, contra the old comment. - tests/observability_quadlet.bats: the two seams now encode the new contract (per-endpoint reachability by process type; every named series must be on the measured list for its engine; engine-independent rules must name both). - homepage docs.html/services.json: frank-medium sees pictures, frank-large is CIRU Orca with a single seat (was halogen w4b, two seats, 200k). - docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified capabilities, rollback, replica-era sections labelled); llama.cpp runbook carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured verdict; Observability records the namespace split and the DNAT lesson; AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook index flipped; both indexes updated; work-log entry added. Also repaired agent-pass damage: prose prepended above frontmatter openers, duplicated runbook bodies, a literal tool-call token pasted into a runbook, and an index destroyed by diff artifacts (restored from HEAD, flips re-applied). Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
This commit is contained in:
1 parent
c2116658f8
commit
08f1c7ad09
13 files changed
+687
-286
No files matched your search
@@ -300,21 +300,23 @@ PAIRS
|
||||
|
||||
@test "prometheus scrapes every LLM endpoint that exports metrics" {
|
||||
# The LLM endpoints are the services whose behaviour this fleet argues about
|
||||
# most (KV pressure, prefix-cache reuse, queue depth), and vLLM exports all
|
||||
# of it. If a scrape target drifts from the endpoint the units publish,
|
||||
# nobody notices until someone needs the graph during an incident.
|
||||
# most (KV pressure, prefix-cache reuse, queue depth). Since frank-medium
|
||||
# moved to radiance/vLLM on 2026-09-20 there are two metric namespaces, and
|
||||
# the three addresses differ by HOW each endpoint is reachable from a
|
||||
# container: netavark DNATs *published* container ports only, so
|
||||
# frank-medium — a rootless Quadlet on another host that publishes 7770 — is
|
||||
# its LAN address, while the two Dankclaw endpoints are HOST processes and
|
||||
# answer only as host.containers.internal. Addressing those two by LAN IP
|
||||
# left both targets red by construction until the 2026-09-20 /targets audit.
|
||||
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
|
||||
grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-medium not scraped"; return 1; }
|
||||
grep -q '"192.168.0.143:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
|
||||
# The medium endpoint is exported by the proxy on the host network; the
|
||||
# quadlet's own contract test pins that port, so name the job too.
|
||||
grep -q 'job_name: "frank-medium"' "$cfg"
|
||||
grep -q 'job_name: "frank-small"' "$cfg"
|
||||
# frank-large is NOT scraped on purpose: the halogen engine serves
|
||||
# /health, /cache and /v1/* and no metrics endpoint, so a target here would
|
||||
# be red by construction and would fire LlmEndpointDown forever. Assert the
|
||||
# absence, so a well-meaning edit cannot re-add a permanently-red probe.
|
||||
! grep -q '192.168.0.143:7770' "$cfg"
|
||||
grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
|
||||
grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; }
|
||||
for job in frank-medium frank-small frank-large; do
|
||||
grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; }
|
||||
done
|
||||
# None of them is reached through a guarded vhost: Prometheus cannot solve
|
||||
# an Anubis challenge, so a franksdank.cloud target is red by construction.
|
||||
! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"
|
||||
}
|
||||
|
||||
@@ -351,23 +353,46 @@ PAIRS
|
||||
import re, yaml
|
||||
d = yaml.safe_load(open('$rules'))
|
||||
names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rules']]
|
||||
for want in ('job:llm_prefix_cache_reuse:rate5m', 'LlmRequestsQueueing',
|
||||
'LlmPrefixCacheCollapse', 'LlmSpecDecodeOff', 'LlmEndpointDown'):
|
||||
for want in ('job:llm_prefix_cache_reuse:rate5m',
|
||||
'job:llm_vllm_prefix_cache_reuse:rate5m',
|
||||
'job:llm_spec_accept_length:rate5m',
|
||||
'LlmRequestsQueueing', 'LlmRequestsWaiting',
|
||||
'LlmKvCacheSaturated', 'LlmPrefixCacheCollapse',
|
||||
'LlmSpecDecodeOff', 'LlmEndpointDown'):
|
||||
assert want in names, want
|
||||
# Every rule must be keyed to a series that exists. The vLLM era's namespaces
|
||||
# are gone (frank-medium is llama.cpp now) and a rule naming one of them
|
||||
# evaluates nothing forever without erroring — the failure this seam exists for.
|
||||
# Every rule must be keyed to a series that actually exists. A rule naming a
|
||||
# series nobody exports evaluates nothing forever and errors never — the failure
|
||||
# this seam exists for. Since 2026-09-20 two engines are scraped, so both
|
||||
# namespaces are legal, but each name must be on the measured list.
|
||||
exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']]
|
||||
assert not any('vllm:' in e for e in exprs), exprs
|
||||
# ...and the llama.cpp series a rule may name are the ones the server exports
|
||||
# (measured from /metrics on 2026-09-16). Anything else is a guess.
|
||||
# llama.cpp families measured from /metrics on 2026-09-16 (frank-small,
|
||||
# frank-large), vLLM families measured from /metrics on 2026-09-20
|
||||
# (frank-medium, radiance/vLLM). Anything else is a guess.
|
||||
live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
|
||||
'llamacpp:requests_deferred', 'llamacpp:requests_processing',
|
||||
'llamacpp:spec_decode_num_drafts_total',
|
||||
'llamacpp:spec_decode_num_accepted_tokens_total',
|
||||
'llamacpp:tokens_predicted_total'}
|
||||
used = {m for e in exprs for m in re.findall(r'llamacpp:[a-z_0-9]+', e)}
|
||||
'llamacpp:tokens_predicted_total',
|
||||
'vllm:kv_cache_usage_perc', 'vllm:num_requests_running',
|
||||
'vllm:num_requests_waiting', 'vllm:num_preemptions_total',
|
||||
'vllm:generation_tokens_total', 'vllm:prompt_tokens_total',
|
||||
'vllm:spec_decode_num_accepted_tokens_total',
|
||||
'vllm:spec_decode_num_draft_tokens_total',
|
||||
'vllm:spec_decode_num_drafts_total',
|
||||
'vllm:prefix_cache_hits_total', 'vllm:prefix_cache_queries_total',
|
||||
'vllm:request_success_total'}
|
||||
used = {m for e in exprs
|
||||
for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', e)}
|
||||
assert used <= live, used - live
|
||||
# Engine-independent rules must cover BOTH families, or the cutover silently
|
||||
# blinds one endpoint: the two rules that describe an engine-agnostic failure
|
||||
# name a series from each namespace.
|
||||
joined = '\n'.join(exprs)
|
||||
for pair in (('LlmSpecDecodeOff', 'llamacpp:spec_decode_num_drafts_total',
|
||||
'vllm:spec_decode_num_drafts_total'),
|
||||
('LlmPrefixCacheCollapse', 'llamacpp:prompt_tokens_total',
|
||||
'vllm:prefix_cache_queries_total')):
|
||||
assert any(pair[1] in e and pair[2] in e for e in exprs), pair[0]
|
||||
print('rules ok:', len(names))
|
||||
"
|
||||
# A rule file that nothing loads evaluates nothing, and Prometheus reports
|
||||
|
||||
Reference in new issue
Block a user