frank-medium: serve the whole 262144-token window, and repoint every signal to llama.cpp

Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.

Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.

The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.

omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.
This commit is contained in:
Frank
2026-09-16 15:29:40 -07:00
parent ab51d61198
commit dc76dc5dc4
8 changed files with 220 additions and 140 deletions
+17 -4
View File
@@ -345,13 +345,26 @@ PAIRS
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
[ -f "$rules" ]
python3 -c "
import yaml
import re, yaml
d = yaml.safe_load(open('$rules'))
names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rules']]
assert 'record' or True
for want in ('LlmKvPoolSaturated', 'LlmRequestsQueueing', 'LlmPrefixCacheCollapse',
'LlmEndpointDown', 'LlmLatencyHigh'):
for want in ('job:llm_prefix_cache_reuse:rate5m', 'LlmRequestsQueueing',
'LlmPrefixCacheCollapse', 'LlmSpecDecodeOff', 'LlmEndpointDown'):
assert want in names, want
# Every rule must be keyed to a series that exists. The vLLM era's namespaces
# are gone (frank-medium is llama.cpp now) and a rule naming one of them
# evaluates nothing forever without erroring — the failure this seam exists for.
exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']]
assert not any('vllm:' in e for e in exprs), exprs
# ...and the llama.cpp series a rule may name are the ones the server exports
# (measured from /metrics on 2026-09-16). Anything else is a guess.
live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
'llamacpp:requests_deferred', 'llamacpp:requests_processing',
'llamacpp:spec_decode_num_drafts_total',
'llamacpp:spec_decode_num_accepted_tokens_total',
'llamacpp:tokens_predicted_total'}
used = {m for e in exprs for m in re.findall(r'llamacpp:[a-z_0-9]+', e)}
assert used <= live, used - live
print('rules ok:', len(names))
"
# A rule file that nothing loads evaluates nothing, and Prometheus reports