frank-primary: 5 x full 262144 contexts at fp8 (final shape)
Frank's call after the night ladder: five contexts at the model's full native 262,144 window beat six shortened ones. Six full-length contexts do NOT fit at any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the 6-seq graph/activation footprint leaves too little for the pool); 5 x 262144 at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured 'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens per request: 5.15x' (39,462 tokens of margin). Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144) re-pinned.
This commit is contained in:
1 parent
7cab15ac39
commit
bd311bf591
6 files changed
+83
-61
No files matched your search
@@ -709,8 +709,17 @@
|
||||
--tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
|
||||
'{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
|
||||
"NO tool dialect" test is flipped to a PRESENT test.
|
||||
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow 262144 ->
|
||||
241664 (compaction gap recomputed: 169,164 + 36,864 <= 241,664) and
|
||||
- **Final shape (Frank's call): 5 seqs x the full native 262144 window at
|
||||
fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB
|
||||
free) — six full-length contexts do not fit at any KV dtype — so the
|
||||
contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first
|
||||
rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens /
|
||||
5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16
|
||||
three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the
|
||||
same night) is the documented max at 3 contexts.
|
||||
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144
|
||||
with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365
|
||||
<= 262,144) and
|
||||
`preserve_thinking` moved from a bare top-level body field to
|
||||
`chat_template_kwargs` (the form vLLM actually reads).
|
||||
- Production bring-up: the winner (241,664) boots and serves on :7770 —
|
||||
|
||||
Reference in new issue
Block a user