From a1b650aabcc482ed1a254a8c86648614307c4d9f Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 1 Oct 2026 13:25:27 -0700 Subject: [PATCH] frank-primary: 3 full windows, pool at the contract floor, MNBT 2048 Frank's call 2026-10-01: MAXLEN 262144 x SEQS 3 = 786,432 tokens of contract. Measured fp8 KV costs 7,698 B/token (4-slot boot 8,311,013,376 B -> 1,079,582 tokens; 5-slot boot 10,388,766,720 B -> 1,350,182 tokens), so the pool floor for the contract is 6.054e9 B. Pinned at 6,100,000,000 (~792k tokens, ~5.9k margin) so the remaining ~2.2 GiB versus the 4-slot shape goes to the expert ARENA - the fork sizes its decode-critical expert cache from the remainder after weights, pool and workspace. MNBT 2048 (Frank): the arena sweep measured 6,499 slots at every rung <= 4096, so the larger batch costs the arena nothing. Seam retargeted (3 seqs / 6.1e9 / MNBT 2048); docs + AGENTS row + log updated, including the closed DCP investigation. --- AGENTS.md | 2 +- scripts/frank-primary.env | 34 +++++++++++-------- ...server-qwen38-fp8-expert-offload-tuning.md | 8 +++-- serverAppConfigs-docs/log.md | 11 ++++++ tests/frank_primary_unit.bats | 22 +++++++----- 5 files changed, 51 insertions(+), 26 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 874c100..0181b19 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **4 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 8,311,013,376/rank, util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Five full-length contexts BOOT but leave ~0 bytes free at runtime — a real client request (995-token prompt, `max_tokens=261149`) killed the worker with `torch.OutOfMemoryError ... 0 bytes free` — so four slots carry the deployed shape and its VRAM headroom. Six do not fit at all (6×241664 asked 10.70 GiB vs 9.18 GiB free). fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 | +| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **3 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 6,100,000,000/rank — the pool pinned to the contract FLOOR so everything else feeds the expert arena — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Five full-length contexts BOOT but leave ~0 bytes free at runtime — a real client request (995-token prompt, `max_tokens=261149`) killed the worker with `torch.OutOfMemoryError ... 0 bytes free`; six never fit at all (6×241664 asked 10.70 GiB vs 9.18 GiB free). Three slots with a floor-sized pool is the deployed shape. DCP cannot help here: it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 | | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 | | Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 4 slots of 151,552, bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 | diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index a39e8ac..819ff38 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -59,22 +59,28 @@ FRANK_PRIMARY_PUBLISH=7770:7770 # hoc experiment identity (both hold the GPUs / name registrations). FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" -# --- qualified engine shape (2026-10-01 night ladder, Frank's method) --------- -# SEQS 4 x MAXLEN 262144 at fp8 KV is the DEPLOYED shape: four contexts of -# the model's FULL native window, with VRAM headroom the five-slot shape did -# not have. Measured 2026-10-01: 5 x 262144 booted (KV 1,350,182 tokens / -# 5.15x) but left ~0 bytes free at runtime, and a real client request -# (995-token prompt, max_tokens=261149) killed the worker with -# `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free`. -# Dropping to four slots returns ~1.93 GiB per rank for activations/graphs. -# At SEQS=3 bf16, 262144/245760 also OOMed at boot and 241664 was max -# possible; at SEQS=6 fp8, six full-length contexts do NOT fit at all -# (6 x 241664 asked 10.70 GiB against 9.18 GiB free). +# --- qualified engine shape (2026-10-01, final: 3 windows + max arena) -------- +# SEQS 3 x MAXLEN 262144 at fp8 KV (Frank's call, 2026-10-01): three contexts +# of the model's FULL native window, with the KV pool pinned to the FLOOR the +# contract needs so that everything else goes to the expert ARENA — the +# remainder is what the fork's planner gives the decode-critical expert cache. +# 3 x 262144 = 786,432 tokens; measured fp8 rate 7,698 B/token (from the +# 4-slot boot: 8,311,013,376 B -> 1,079,582 tokens, and the 5-slot boot: +# 10,388,766,720 B -> 1,350,182 tokens) => 6.054e9 B is the floor. Pinned at +# 6,100,000,000 => ~792k tokens, a ~5.9k-token margin, and ~2.2 GiB returned +# to the arena versus the 4-slot shape. +# History in this shape: 5 x 262144 booted (KV 1,350,182 / 5.15x) but left 0 +# bytes free and a real request (995-token prompt, max_tokens=261149) killed +# the worker; 4 x 262144 (KV 1,079,582 / 4.12x) was the runtime-survivable +# stopgap. Six full-length contexts never fit (6 x 241664 asked 10.70 GiB +# against 9.18 GiB free). +# MNBT 2048 (Frank's call): the arena sweep measured it at the same 6,499-slot +# ceiling as <=4096, so the bigger batch costs the arena nothing. FRANK_PRIMARY_MAXLEN=262144 -FRANK_PRIMARY_SEQS=4 -FRANK_PRIMARY_KV_BYTES=8311013376 +FRANK_PRIMARY_SEQS=3 +FRANK_PRIMARY_KV_BYTES=6100000000 FRANK_PRIMARY_KV_DTYPE=fp8 -FRANK_PRIMARY_MNBT=1024 +FRANK_PRIMARY_MNBT=2048 FRANK_PRIMARY_GATHER_CHUNKS=32 FRANK_PRIMARY_GATHER_LANES=64 FRANK_PRIMARY_UTIL=0.98 diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index 81f3422..ba0de8c 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -278,8 +278,9 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap. ### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec) ``` -FRANK_PRIMARY_MAXLEN=262144 x SEQS 4 (1,048,576 tokens of contract) -FRANK_PRIMARY_KV_BYTES=8311013376 PER RANK (fp8, pool scales with windows) +FRANK_PRIMARY_MAXLEN=262144 x SEQS 3 (786,432 tokens of contract) +FRANK_PRIMARY_KV_BYTES=6100000000 PER RANK (fp8, pinned to the CONTRACT + FLOOR so the remainder feeds the arena) FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (K and V together) FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env) @@ -298,7 +299,8 @@ memory environment). The night searched two concurrency contracts: | **3 × 241664, bf16** | **PASS — KV 732,359 tokens, 3.03x** (max possible at 3 contexts) | | 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit | | 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x | -| **4 × 262144, fp8** | **DEPLOYED.** Five slots booted but left 0 bytes free at runtime: a real client request (995-token prompt, `max_tokens=261149`) killed the worker (`Tried to allocate 50.00 MiB ... 0 bytes free`). Dropping one slot returns ~1.93 GiB per rank for activations and graph capture | +| 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x | +| **3 × 262144, fp8, pool at the floor** | **DEPLOYED** (Frank's call): 786,432 tokens of contract; pool pinned at 6,100,000,000 B/rank (the measured 7,698 B/token floor is 6.054e9) so the ~2.2 GiB difference versus the 4-slot shape goes to the expert ARENA instead of unused KV | Boot is NOT proof of runtime survivability: the five-slot shape passed every boot gate and still died on the first long-generation request. After any diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 1eaa5d0..b865a08 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -737,6 +737,17 @@ "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed down until a human noticed. An explicit `systemctl stop` still wins, so bench windows and deploys are unaffected. +- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR, + MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of + contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor + and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing + the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the + decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the + arena nothing (the sweep measured 6,499 slots at every rung <= 4096). +- **DCP investigated and closed** (see the parallelism survey in the runbook): + `--decode-context-parallel-size 2` is rejected at boot — DCP for GQA requires + TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning + knob on this box. - Production bring-up: the shape boots and serves on :7770 — engine log verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live contract probes (reasoning split, tool_calls). Rollback target for the diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index ce3ed92..f0fc669 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -89,12 +89,16 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # deployed shape: ~1.93 GiB more headroom per rank for # activations/graphs. UTIL stays 0.98 (lowering it shrinks the engine's # own budget instead of buying slack). + # 3 x 262144 = 786,432 tokens; at the measured 7,698 B/token the pool + # floor is 6.054e9 B, pinned at 6.1e9 so the margin is ~5.9k tokens and + # EVERYTHING ELSE goes to the expert arena (the decode-critical expert + # cache is the remainder after weights, pool and workspace). grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_SEQS=4$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_KV_BYTES=8311013376$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_KV_BYTES=6100000000$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE" ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE" @@ -182,11 +186,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE" } -@test "MNBT is locked at 1024" { - # Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); - # 1024-4096 are interchangeable on decode; 12288 costs 2.5% plus arena. - # 1024 keeps the workspace floor so the expert arena stays max. - grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE" +@test "MNBT is at Frank's 2048 with the arena intact" { + # MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are + # interchangeable on decode AND identical on arena (the sweep measured + # 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena. + # 2048 is Frank's setting as of 2026-10-01 — it keeps the workspace floor + # so the expert arena stays max. + grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE" grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE" }