frank-primary: 5 x full 262144 contexts at fp8 (final shape)
Frank's call after the night ladder: five contexts at the model's full native 262,144 window beat six shortened ones. Six full-length contexts do NOT fit at any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the 6-seq graph/activation footprint leaves too little for the pool); 5 x 262144 at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured 'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens per request: 5.15x' (39,462 tokens of margin). Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144) re-pinned.
This commit is contained in:
1 parent
7cab15ac39
commit
bd311bf591
6 files changed
+83
-61
No files matched your search
@@ -4,9 +4,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-01):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, 241664 ctx x 3 seqs
|
||||
(max possible — 262144 does not fit at a 3-seq contract on this box),
|
||||
bf16 KV, no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **5 seqs x 262144 ctx
|
||||
(the model's full native window)**, fp8 KV (6 full-length contexts do not
|
||||
fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
@@ -32,8 +32,8 @@ gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
|
||||
TP=2, expert offload automatic, 241664 ctx x 3,
|
||||
bf16 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
TP=2, expert offload automatic, 5 x 262144 ctx,
|
||||
fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
|
||||
Reference in new issue
Block a user