feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets

The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.

- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
  LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
  had), the job:llm_vllm_prefix_cache_reuse:rate5m and
  job:llm_spec_accept_length:rate5m recording rules, and re-cut
  LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
  per-family traffic floors so an idle endpoint cannot fire them. Every
  expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
  and frank-large scraped at host.containers.internal:7770. Both are HOST
  processes on Dankclaw and netavark DNATs published container ports only, so
  the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
  build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
  (per-endpoint reachability by process type; every named series must be on the
  measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
  CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
  capabilities, rollback, replica-era sections labelled); llama.cpp runbook
  carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
  verdict; Observability records the namespace split and the DNAT lesson;
  AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
  index flipped; both indexes updated; work-log entry added.

Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
This commit is contained in:
Frank committed 2026-09-20 19:35:56 -07:00
1 parent c2116658f8
commit 08f1c7ad09
13 files changed
+687 -286

No files matched your search

+48 -23
View File
@@ -300,21 +300,23 @@ PAIRS
@test "prometheus scrapes every LLM endpoint that exports metrics" {
# The LLM endpoints are the services whose behaviour this fleet argues about
# most (KV pressure, prefix-cache reuse, queue depth), and vLLM exports all
# of it. If a scrape target drifts from the endpoint the units publish,
# nobody notices until someone needs the graph during an incident.
# most (KV pressure, prefix-cache reuse, queue depth). Since frank-medium
# moved to radiance/vLLM on 2026-09-20 there are two metric namespaces, and
# the three addresses differ by HOW each endpoint is reachable from a
# container: netavark DNATs *published* container ports only, so
# frank-medium — a rootless Quadlet on another host that publishes 7770 — is
# its LAN address, while the two Dankclaw endpoints are HOST processes and
# answer only as host.containers.internal. Addressing those two by LAN IP
# left both targets red by construction until the 2026-09-20 /targets audit.
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-medium not scraped"; return 1; }
grep -q '"192.168.0.143:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
# The medium endpoint is exported by the proxy on the host network; the
# quadlet's own contract test pins that port, so name the job too.
grep -q 'job_name: "frank-medium"' "$cfg"
grep -q 'job_name: "frank-small"' "$cfg"
# frank-large is NOT scraped on purpose: the halogen engine serves
# /health, /cache and /v1/* and no metrics endpoint, so a target here would
# be red by construction and would fire LlmEndpointDown forever. Assert the
# absence, so a well-meaning edit cannot re-add a permanently-red probe.
! grep -q '192.168.0.143:7770' "$cfg"
grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; }
for job in frank-medium frank-small frank-large; do
grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; }
done
# None of them is reached through a guarded vhost: Prometheus cannot solve
# an Anubis challenge, so a franksdank.cloud target is red by construction.
! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"
}
@@ -351,23 +353,46 @@ PAIRS
import re, yaml
d = yaml.safe_load(open('$rules'))
names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rules']]
for want in ('job:llm_prefix_cache_reuse:rate5m', 'LlmRequestsQueueing',
'LlmPrefixCacheCollapse', 'LlmSpecDecodeOff', 'LlmEndpointDown'):
for want in ('job:llm_prefix_cache_reuse:rate5m',
'job:llm_vllm_prefix_cache_reuse:rate5m',
'job:llm_spec_accept_length:rate5m',
'LlmRequestsQueueing', 'LlmRequestsWaiting',
'LlmKvCacheSaturated', 'LlmPrefixCacheCollapse',
'LlmSpecDecodeOff', 'LlmEndpointDown'):
assert want in names, want
# Every rule must be keyed to a series that exists. The vLLM era's namespaces
# are gone (frank-medium is llama.cpp now) and a rule naming one of them
# evaluates nothing forever without erroring — the failure this seam exists for.
# Every rule must be keyed to a series that actually exists. A rule naming a
# series nobody exports evaluates nothing forever and errors never — the failure
# this seam exists for. Since 2026-09-20 two engines are scraped, so both
# namespaces are legal, but each name must be on the measured list.
exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']]
assert not any('vllm:' in e for e in exprs), exprs
# ...and the llama.cpp series a rule may name are the ones the server exports
# (measured from /metrics on 2026-09-16). Anything else is a guess.
# llama.cpp families measured from /metrics on 2026-09-16 (frank-small,
# frank-large), vLLM families measured from /metrics on 2026-09-20
# (frank-medium, radiance/vLLM). Anything else is a guess.
live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
'llamacpp:requests_deferred', 'llamacpp:requests_processing',
'llamacpp:spec_decode_num_drafts_total',
'llamacpp:spec_decode_num_accepted_tokens_total',
'llamacpp:tokens_predicted_total'}
used = {m for e in exprs for m in re.findall(r'llamacpp:[a-z_0-9]+', e)}
'llamacpp:tokens_predicted_total',
'vllm:kv_cache_usage_perc', 'vllm:num_requests_running',
'vllm:num_requests_waiting', 'vllm:num_preemptions_total',
'vllm:generation_tokens_total', 'vllm:prompt_tokens_total',
'vllm:spec_decode_num_accepted_tokens_total',
'vllm:spec_decode_num_draft_tokens_total',
'vllm:spec_decode_num_drafts_total',
'vllm:prefix_cache_hits_total', 'vllm:prefix_cache_queries_total',
'vllm:request_success_total'}
used = {m for e in exprs
for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', e)}
assert used <= live, used - live
# Engine-independent rules must cover BOTH families, or the cutover silently
# blinds one endpoint: the two rules that describe an engine-agnostic failure
# name a series from each namespace.
joined = '\n'.join(exprs)
for pair in (('LlmSpecDecodeOff', 'llamacpp:spec_decode_num_drafts_total',
'vllm:spec_decode_num_drafts_total'),
('LlmPrefixCacheCollapse', 'llamacpp:prompt_tokens_total',
'vllm:prefix_cache_queries_total')):
assert any(pair[1] in e and pair[2] in e for e in exprs), pair[0]
print('rules ok:', len(names))
"
# A rule file that nothing loads evaluates nothing, and Prometheus reports