frank-primary: max-possible window (241664) + reasoning/tool contract
Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.
Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.
Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
This commit is contained in:
1 parent
e8524d4046
commit
7cab15ac39
7 files changed
+210
-53
No files matched your search
@@ -1,17 +1,21 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-30):
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-01):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
Intel/GLM-5.3-Flash-W4A16-AutoRound — symmetric int4 group 128 — across both
|
||||
Radeon AI PRO R9700s on :7770, TP=2 with the expert set offloaded to 73.64 GiB
|
||||
of pinned host RAM per rank behind a 14.51 GiB VRAM LRU arena; 131,072 ctx,
|
||||
bf16 KV, vision and tool calls on, measured 9.9-10.0 tok/s decode), and
|
||||
Dankclaw runs two llama.cpp endpoints —
|
||||
stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
|
||||
Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
|
||||
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
|
||||
QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, 241664 ctx x 3 seqs
|
||||
(max possible — 262144 does not fit at a 3-seq contract on this box),
|
||||
bf16 KV, no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
runs two endpoints —
|
||||
gufo v0.1.1 (the Strix Halo engine, built from source; frank-large,
|
||||
Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M
|
||||
drafter on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
|
||||
HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090,
|
||||
port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -27,9 +31,9 @@ graph LR
|
||||
gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
podman run, frank-primary, port 7770]
|
||||
C[GLM-5.3-Flash W4A16 int4
|
||||
TP=2, expert offload 73.64 GiB/rank pinned,
|
||||
14.51 GiB LRU arena, 131072 ctx, vision, tools]
|
||||
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
|
||||
TP=2, expert offload automatic, 241664 ctx x 3,
|
||||
bf16 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
|
||||
Reference in new issue
Block a user