frank-medium: serve the whole 262144-token window, and repoint every signal to llama.cpp
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and 29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a 140,069-token prompt prefilled through the single slot in 134 s to prove the window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this runtime's. Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to 140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance, and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt. The signals move with the runtime, so the fleet's rules and dashboard are re-cut rather than renamed: vllm: series do not exist any more, and two of the old rules watched things llama.cpp does not export at all (KV-pool fill, p95 latency). Those are retired with the reason written down, and LlmSpecDecodeOff takes one of the slots -- a lost drafter is the failure no client can see. The observability seam now fails if any rule names a series the endpoint does not export, which is what would have caught the stale vllm: expressions. omp models.yml (outside the repo) is updated in the same pass: contextWindow 262144, and the full reasoning-effort map, because Qwen3.8's template answers 500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer 200. Subagents run at high.
This commit is contained in:
1 parent
ab51d61198
commit
dc76dc5dc4
8 files changed
+220
-140
No files matched your search
@@ -100,12 +100,19 @@ DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter
|
||||
[ "$parallel" -ge 1 ]
|
||||
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
|
||||
# qwen35 spends KV on its 16 full-attention layers only: 2 (K,V) x 4 heads
|
||||
# x 256 dim x 2 B = 64 KiB per token at f16. Worst case for the fit is ONE
|
||||
# card carrying the drafter and every attention layer's KV, with ~2 GiB of
|
||||
# compute buffers: half the weights, plus that, must stay inside the card.
|
||||
# x 256 dim x 2 B = 64 KiB per token at f16. A layer split puts those
|
||||
# layers (and their KV) on both cards, so the worst card carries half the
|
||||
# weights, half the KV, the drafter and ~2 GiB of compute buffers.
|
||||
#
|
||||
# This model is deliberately calibrated against rocm-smi: at 262144 it
|
||||
# predicts 28.53e9 B against the 29.14e9 B measured on the busier card
|
||||
# (2026-09-16), i.e. ~2 % optimistic. Keep it that way -- a model that is
|
||||
# pessimistic beyond a few GiB would reject a context that fits.
|
||||
local per_card
|
||||
per_card=$(( MODEL_BYTES / 2 + DRAFT_BYTES + ctx * 65536 + 2 * 1024 * 1024 * 1024 ))
|
||||
per_card=$(( MODEL_BYTES / 2 + DRAFT_BYTES + (ctx * 65536) / 2 + 2 * 1024 * 1024 * 1024 ))
|
||||
[ "$per_card" -le "$CARD_BYTES" ]
|
||||
# ...and the window may not exceed what the checkpoint was trained for.
|
||||
[ "$ctx" -le 262144 ]
|
||||
}
|
||||
|
||||
@test "frank-medium is text only: the vision tower stays on frank-small" {
|
||||
|
||||
Reference in new issue
Block a user