frank-primary: max-possible window (241664) + reasoning/tool contract

Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.

Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.

Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
This commit is contained in:
Frank committed 2026-10-01 03:08:52 -07:00
1 parent e8524d4046
commit 7cab15ac39
7 files changed
+210 -53

No files matched your search

+17 -13
View File
@@ -1,17 +1,21 @@
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on the two hosts (2026-09-30):
Documentation for the LLM inference fleet on the two hosts (2026-10-01):
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
Intel/GLM-5.3-Flash-W4A16-AutoRound — symmetric int4 group 128 — across both
Radeon AI PRO R9700s on :7770, TP=2 with the expert set offloaded to 73.64 GiB
of pinned host RAM per rank behind a 14.51 GiB VRAM LRU arena; 131,072 ctx,
bf16 KV, vision and tool calls on, measured 9.9-10.0 tok/s decode), and
Dankclaw runs two llama.cpp endpoints —
stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
on :7770, TP=2 with automatic tiered-expert offload, 241664 ctx x 3 seqs
(max possible — 262144 does not fit at a 3-seq contract on this box),
bf16 KV, no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
runs two endpoints —
gufo v0.1.1 (the Strix Halo engine, built from source; frank-large,
Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M
drafter on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090,
port 7771).
> 2026-08-22: the former `vllmScripts` repository is merged into the
> serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -27,9 +31,9 @@ graph LR
gfx1201, 32 GiB each]
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
podman run, frank-primary, port 7770]
C[GLM-5.3-Flash W4A16 int4
TP=2, expert offload 73.64 GiB/rank pinned,
14.51 GiB LRU arena, 131072 ctx, vision, tools]
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
TP=2, expert offload automatic, 241664 ctx x 3,
bf16 KV, MNBT 1024, gather 32x64, qwen3 parsers]
end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]