frank-primary: max-possible window (241664) + reasoning/tool contract

Night ladder result (Frank's subtract-16k method): 262144 and 245760 both OOM
at CUDA graph capture ('Tried to allocate 76.00 MiB ... 0 bytes free'), 237568
passed (KV 718,606 tokens), and the +4096 bonus step PASSED -> 241664 is max
possible and DEPLOYED at util 0.98 (KV 732,359 tokens / 3.03x concurrency at
241,664-token requests). Util stays 0.98: lowering it shrinks the engine's own
budget instead of buying slack.

Client contract restored (the GLM rollback d91379d dropped all three and the
qwen rollback inherited the gap): --reasoning-parser qwen3 fixes the malformed
thinking block (thinking was landing inline in content with a dangling
</think> and a null reasoning field), --enable-auto-tool-choice with
--tool-call-parser qwen3_xml makes tools work, and
--default-chat-template-kwargs enables enable_thinking + preserve_thinking.

Harness: contextWindow 241664 (compaction gap recomputed), preserve_thinking
moved into chat_template_kwargs.
This commit is contained in:
Frank committed 2026-10-01 03:08:52 -07:00
1 parent e8524d4046
commit 7cab15ac39
7 files changed
+210 -53

No files matched your search

+1 -1
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload behind a 15.05 GiB / 6,574-slot VRAM LRU arena, MTP4, prefetch OFF, 200000 ctx × 3 seqs, fp8 KV; measured 37.4-40.6 tok/s decode / ~54 % acceptance (the fleet's only endpoint meeting the 35 target). Rolled back 2026-10-01 from the failed GLM/DFlash2 experiment (commit `2339fae` trio) | 7770 |
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **241664 ctx × 3 seqs** at KV_BYTES 10,184,000,000/rank (measured pool 732,359 tokens / 3.03× concurrency), util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. 241,664 is **max possible** on this box (Frank's subtract-16k ladder: 262144/245760 OOM at graph capture, 237568 passed, +4096 bonus step won). fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |