retire frank-medium: tests, planning and observability ground on frank-primary

The frank-medium retirement tail the 2026-09-26 lucebox cutover left
uncommitted: the radiance quadlet tree, its swap/fetch tooling and its
seam test are deleted (archive copies live in dankserver/.AGENTTRASH/
20260926-frank-medium-radiance/ and scripts/.AGENTTRASH/20260926-
frank-medium-radiance/), the planner/allowlist/test-changed mappings
drop the service, and frank-large's chat template moves to scripts/
(its old home was the retired frank-medium dir).

Finished here:
- tests/frank_primary_unit.bats: the lucebox quadlet seam (boot
  persistence, host-network bind, HIP_VISIBLE_DEVICES=0,1, HIP-reserved
  env-name guard, checkout-mounted entrypoint/healthcheck, first-start
  patience, deployer payload/port gates).
- test-changed.sh: explicit .AGENTTRASH/ case — retirement commits move
  files to trash, and the wrapper refused those additions with "no
  focused-test mapping".
- observability: the vllm: namespace left with the radiance runtime.
  Rules keep only llamacpp:* series (prefix-cache reuse, spec-decode,
  queueing, endpoint-down); the dashboard drops the vLLM KV/latency
  panels; prometheus.yml documents that frank-primary (lucebox) has no
  /metrics and therefore no scrape job. llm-fleet.json was left
  corrupted mid-edit (missing opening brace) and is rewritten from HEAD.

Suite: 339 green (was 7 red).
This commit is contained in:
Frank
2026-09-26 19:25:46 -07:00
parent 32d264ccda
commit d7bf9f119a
22 changed files with 243 additions and 2001 deletions
+26 -36
View File
@@ -315,19 +315,24 @@ PAIRS
@test "prometheus scrapes every LLM endpoint that exports metrics" {
# The LLM endpoints are the services whose behaviour this fleet argues about
# most (KV pressure, prefix-cache reuse, queue depth). Since frank-medium
# moved to radiance/vLLM on 2026-09-20 there are two metric namespaces, and
# the three addresses differ by HOW each endpoint is reachable from a
# container: netavark DNATs *published* container ports only, so
# frank-medium — a rootless Quadlet on another host that publishes 7770 — is
# its LAN address, while the two Dankclaw endpoints are HOST processes and
# answer only as host.containers.internal. Addressing those two by LAN IP
# left both targets red by construction until the 2026-09-20 /targets audit.
# most (KV pressure, prefix-cache reuse, queue depth). Only the two Dankclaw
# llama.cpp endpoints export metrics today: frank-primary (Dankserver, the
# Lucebox DS4.1 engine since the 2026-09-26 cutover) answers 404 on
# /metrics, so a job for it would be a permanently-down target — no job,
# and its LAN address must not sneak back in with one.
# The two scraped endpoints are HOST processes reached through
# host.containers.internal: this Prometheus is a bridge-networked
# container, and on this netavark host a container can only reach the
# host's LAN IP at a PUBLISHED port — a host-process bind like
# llama-server's 0.0.0.0:7770 is refused at 192.168.0.143:7770 while it
# answers at 169.254.1.2:7770 (why frank-small stayed red by construction
# for days until the 2026-09-20 /targets audit).
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-medium not scraped"; return 1; }
! grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-primary is scraped but exports no metrics"; return 1; }
grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; }
for job in frank-medium frank-small frank-large; do
for job in frank-small frank-large; do
grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; }
done
# None of them is reached through a guarded vhost: Prometheus cannot solve
@@ -369,45 +374,30 @@ import re, yaml
d = yaml.safe_load(open('$rules'))
names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rules']]
for want in ('job:llm_prefix_cache_reuse:rate5m',
'job:llm_vllm_prefix_cache_reuse:rate5m',
'job:llm_spec_accept_length:rate5m',
'LlmRequestsQueueing', 'LlmRequestsWaiting',
'LlmKvCacheSaturated', 'LlmPrefixCacheCollapse',
'LlmRequestsQueueing', 'LlmPrefixCacheCollapse',
'LlmSpecDecodeOff', 'LlmEndpointDown'):
assert want in names, want
# Every rule must be keyed to a series that actually exists. A rule naming a
# series nobody exports evaluates nothing forever and errors never — the failure
# this seam exists for. Since 2026-09-20 two engines are scraped, so both
# namespaces are legal, but each name must be on the measured list.
# series nobody exports evaluates nothing forever and errors never — the
# failure this seam exists for. Since the 2026-09-26 cutover the only scraped
# LLM runtime is llama.cpp (frank-small, frank-large); the vllm: family left
# with frank-medium's radiance runtime and must not survive in any expr.
exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']]
# llama.cpp families measured from /metrics on 2026-09-16 (frank-small,
# frank-large), vLLM families measured from /metrics on 2026-09-20
# (frank-medium, radiance/vLLM). Anything else is a guess.
# llama.cpp families measured from /metrics on 2026-09-16 and 2026-09-20
# (frank-small, frank-large). Anything else is a guess.
live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
'llamacpp:requests_deferred', 'llamacpp:requests_processing',
'llamacpp:spec_decode_num_drafts_total',
'llamacpp:spec_decode_num_accepted_tokens_total',
'llamacpp:spec_decode_num_draft_tokens_total',
'llamacpp:tokens_predicted_total',
'vllm:kv_cache_usage_perc', 'vllm:num_requests_running',
'vllm:num_requests_waiting', 'vllm:num_preemptions_total',
'vllm:generation_tokens_total', 'vllm:prompt_tokens_total',
'vllm:spec_decode_num_accepted_tokens_total',
'vllm:spec_decode_num_draft_tokens_total',
'vllm:spec_decode_num_drafts_total',
'vllm:prefix_cache_hits_total', 'vllm:prefix_cache_queries_total',
'vllm:request_success_total'}
'llamacpp:predicted_tokens_seconds', 'llamacpp:prompt_tokens_seconds',
'llamacpp:n_decode_total', 'llamacpp:n_busy_slots_per_decode'}
used = {m for e in exprs
for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', e)}
assert used <= live, used - live
# Engine-independent rules must cover BOTH families, or the cutover silently
# blinds one endpoint: the two rules that describe an engine-agnostic failure
# name a series from each namespace.
joined = '\n'.join(exprs)
for pair in (('LlmSpecDecodeOff', 'llamacpp:spec_decode_num_drafts_total',
'vllm:spec_decode_num_drafts_total'),
('LlmPrefixCacheCollapse', 'llamacpp:prompt_tokens_total',
'vllm:prefix_cache_queries_total')):
assert any(pair[1] in e and pair[2] in e for e in exprs), pair[0]
assert not any(':vllm' in e for e in exprs), 'vllm: series in rules but no vllm endpoint is scraped'
print('rules ok:', len(names))
"
# A rule file that nothing loads evaluates nothing, and Prometheus reports