frank-primary: 5 x full 262144 contexts at fp8 (final shape)

Frank's call after the night ladder: five contexts at the model's full native
262,144 window beat six shortened ones. Six full-length contexts do NOT fit at
any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the
6-seq graph/activation footprint leaves too little for the pool); 5 x 262144
at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured
'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens
per request: 5.15x' (39,462 tokens of margin).

Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured
earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling
law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144)
re-pinned.
This commit is contained in:
Frank committed 2026-10-01 03:52:10 -07:00
1 parent 7cab15ac39
commit bd311bf591
6 files changed
+83 -61

No files matched your search

+5 -5
View File
@@ -4,9 +4,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-01):
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
on :7770, TP=2 with automatic tiered-expert offload, 241664 ctx x 3 seqs
(max possible — 262144 does not fit at a 3-seq contract on this box),
bf16 KV, no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
on :7770, TP=2 with automatic tiered-expert offload, **5 seqs x 262144 ctx
(the model's full native window)**, fp8 KV (6 full-length contexts do not
fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
@@ -32,8 +32,8 @@ gfx1201, 32 GiB each]
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
podman run, frank-primary, port 7770]
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
TP=2, expert offload automatic, 241664 ctx x 3,
bf16 KV, MNBT 1024, gather 32x64, qwen3 parsers]
TP=2, expert offload automatic, 5 x 262144 ctx,
fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]