frank-primary: 5 x full 262144 contexts at fp8 (final shape)

Frank's call after the night ladder: five contexts at the model's full native
262,144 window beat six shortened ones. Six full-length contexts do NOT fit at
any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the
6-seq graph/activation footprint leaves too little for the pool); 5 x 262144
at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured
'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens
per request: 5.15x' (39,462 tokens of margin).

Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured
earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling
law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144)
re-pinned.
This commit is contained in:
Frank committed 2026-10-01 03:52:10 -07:00
1 parent 7cab15ac39
commit bd311bf591
6 files changed
+83 -61

No files matched your search

+11 -2
View File
@@ -709,8 +709,17 @@
--tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
'{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
"NO tool dialect" test is flipped to a PRESENT test.
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow 262144 ->
241664 (compaction gap recomputed: 169,164 + 36,864 <= 241,664) and
- **Final shape (Frank's call): 5 seqs x the full native 262144 window at
fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB
free) — six full-length contexts do not fit at any KV dtype — so the
contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first
rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens /
5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16
three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the
same night) is the documented max at 3 contexts.
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144
with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365
<= 262,144) and
`preserve_thinking` moved from a bare top-level body field to
`chat_template_kwargs` (the form vLLM actually reads).
- Production bring-up: the winner (241,664) boots and serves on :7770 —