retire frank-medium: tests, planning and observability ground on frank-primary
The frank-medium retirement tail the 2026-09-26 lucebox cutover left uncommitted: the radiance quadlet tree, its swap/fetch tooling and its seam test are deleted (archive copies live in dankserver/.AGENTTRASH/ 20260926-frank-medium-radiance/ and scripts/.AGENTTRASH/20260926- frank-medium-radiance/), the planner/allowlist/test-changed mappings drop the service, and frank-large's chat template moves to scripts/ (its old home was the retired frank-medium dir). Finished here: - tests/frank_primary_unit.bats: the lucebox quadlet seam (boot persistence, host-network bind, HIP_VISIBLE_DEVICES=0,1, HIP-reserved env-name guard, checkout-mounted entrypoint/healthcheck, first-start patience, deployer payload/port gates). - test-changed.sh: explicit .AGENTTRASH/ case — retirement commits move files to trash, and the wrapper refused those additions with "no focused-test mapping". - observability: the vllm: namespace left with the radiance runtime. Rules keep only llamacpp:* series (prefix-cache reuse, spec-decode, queueing, endpoint-down); the dashboard drops the vLLM KV/latency panels; prometheus.yml documents that frank-primary (lucebox) has no /metrics and therefore no scrape job. llm-fleet.json was left corrupted mid-edit (missing opening brace) and is rewritten from HEAD. Suite: 339 green (was 7 red).
This commit is contained in:
@@ -315,19 +315,24 @@ PAIRS
|
||||
|
||||
@test "prometheus scrapes every LLM endpoint that exports metrics" {
|
||||
# The LLM endpoints are the services whose behaviour this fleet argues about
|
||||
# most (KV pressure, prefix-cache reuse, queue depth). Since frank-medium
|
||||
# moved to radiance/vLLM on 2026-09-20 there are two metric namespaces, and
|
||||
# the three addresses differ by HOW each endpoint is reachable from a
|
||||
# container: netavark DNATs *published* container ports only, so
|
||||
# frank-medium — a rootless Quadlet on another host that publishes 7770 — is
|
||||
# its LAN address, while the two Dankclaw endpoints are HOST processes and
|
||||
# answer only as host.containers.internal. Addressing those two by LAN IP
|
||||
# left both targets red by construction until the 2026-09-20 /targets audit.
|
||||
# most (KV pressure, prefix-cache reuse, queue depth). Only the two Dankclaw
|
||||
# llama.cpp endpoints export metrics today: frank-primary (Dankserver, the
|
||||
# Lucebox DS4.1 engine since the 2026-09-26 cutover) answers 404 on
|
||||
# /metrics, so a job for it would be a permanently-down target — no job,
|
||||
# and its LAN address must not sneak back in with one.
|
||||
# The two scraped endpoints are HOST processes reached through
|
||||
# host.containers.internal: this Prometheus is a bridge-networked
|
||||
# container, and on this netavark host a container can only reach the
|
||||
# host's LAN IP at a PUBLISHED port — a host-process bind like
|
||||
# llama-server's 0.0.0.0:7770 is refused at 192.168.0.143:7770 while it
|
||||
# answers at 169.254.1.2:7770 (why frank-small stayed red by construction
|
||||
# for days until the 2026-09-20 /targets audit).
|
||||
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
|
||||
grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-medium not scraped"; return 1; }
|
||||
! grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-primary is scraped but exports no metrics"; return 1; }
|
||||
|
||||
grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
|
||||
grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; }
|
||||
for job in frank-medium frank-small frank-large; do
|
||||
for job in frank-small frank-large; do
|
||||
grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; }
|
||||
done
|
||||
# None of them is reached through a guarded vhost: Prometheus cannot solve
|
||||
@@ -369,45 +374,30 @@ import re, yaml
|
||||
d = yaml.safe_load(open('$rules'))
|
||||
names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rules']]
|
||||
for want in ('job:llm_prefix_cache_reuse:rate5m',
|
||||
'job:llm_vllm_prefix_cache_reuse:rate5m',
|
||||
'job:llm_spec_accept_length:rate5m',
|
||||
'LlmRequestsQueueing', 'LlmRequestsWaiting',
|
||||
'LlmKvCacheSaturated', 'LlmPrefixCacheCollapse',
|
||||
'LlmRequestsQueueing', 'LlmPrefixCacheCollapse',
|
||||
'LlmSpecDecodeOff', 'LlmEndpointDown'):
|
||||
assert want in names, want
|
||||
# Every rule must be keyed to a series that actually exists. A rule naming a
|
||||
# series nobody exports evaluates nothing forever and errors never — the failure
|
||||
# this seam exists for. Since 2026-09-20 two engines are scraped, so both
|
||||
# namespaces are legal, but each name must be on the measured list.
|
||||
# series nobody exports evaluates nothing forever and errors never — the
|
||||
# failure this seam exists for. Since the 2026-09-26 cutover the only scraped
|
||||
# LLM runtime is llama.cpp (frank-small, frank-large); the vllm: family left
|
||||
# with frank-medium's radiance runtime and must not survive in any expr.
|
||||
exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']]
|
||||
# llama.cpp families measured from /metrics on 2026-09-16 (frank-small,
|
||||
# frank-large), vLLM families measured from /metrics on 2026-09-20
|
||||
# (frank-medium, radiance/vLLM). Anything else is a guess.
|
||||
# llama.cpp families measured from /metrics on 2026-09-16 and 2026-09-20
|
||||
# (frank-small, frank-large). Anything else is a guess.
|
||||
live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
|
||||
'llamacpp:requests_deferred', 'llamacpp:requests_processing',
|
||||
'llamacpp:spec_decode_num_drafts_total',
|
||||
'llamacpp:spec_decode_num_accepted_tokens_total',
|
||||
'llamacpp:spec_decode_num_draft_tokens_total',
|
||||
'llamacpp:tokens_predicted_total',
|
||||
'vllm:kv_cache_usage_perc', 'vllm:num_requests_running',
|
||||
'vllm:num_requests_waiting', 'vllm:num_preemptions_total',
|
||||
'vllm:generation_tokens_total', 'vllm:prompt_tokens_total',
|
||||
'vllm:spec_decode_num_accepted_tokens_total',
|
||||
'vllm:spec_decode_num_draft_tokens_total',
|
||||
'vllm:spec_decode_num_drafts_total',
|
||||
'vllm:prefix_cache_hits_total', 'vllm:prefix_cache_queries_total',
|
||||
'vllm:request_success_total'}
|
||||
'llamacpp:predicted_tokens_seconds', 'llamacpp:prompt_tokens_seconds',
|
||||
'llamacpp:n_decode_total', 'llamacpp:n_busy_slots_per_decode'}
|
||||
used = {m for e in exprs
|
||||
for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', e)}
|
||||
assert used <= live, used - live
|
||||
# Engine-independent rules must cover BOTH families, or the cutover silently
|
||||
# blinds one endpoint: the two rules that describe an engine-agnostic failure
|
||||
# name a series from each namespace.
|
||||
joined = '\n'.join(exprs)
|
||||
for pair in (('LlmSpecDecodeOff', 'llamacpp:spec_decode_num_drafts_total',
|
||||
'vllm:spec_decode_num_drafts_total'),
|
||||
('LlmPrefixCacheCollapse', 'llamacpp:prompt_tokens_total',
|
||||
'vllm:prefix_cache_queries_total')):
|
||||
assert any(pair[1] in e and pair[2] in e for e in exprs), pair[0]
|
||||
assert not any(':vllm' in e for e in exprs), 'vllm: series in rules but no vllm endpoint is scraped'
|
||||
print('rules ok:', len(names))
|
||||
"
|
||||
# A rule file that nothing loads evaluates nothing, and Prometheus reports
|
||||
|
||||
Reference in New Issue
Block a user