From 295d5784d3ed00b03fdf40ba9644f20e734453a0 Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 2 Oct 2026 11:48:09 -0700 Subject: [PATCH] frank-primary: record the measured shape-D numbers (398,658 tokens, 1.99x, 6,496 slots) Comments/docs only - no functional change. The live boot of c769ff2 reported: GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99x expert cache space 14.87 GiB / 6,496 slots of 2.344 MiB The arena prediction (6,496) landed exactly; the concurrency prediction (2.04x) did not, because the per-token KV price is SHAPE-DEPENDENT: 7,701 B/token at MAXLEN 262144 (6.1e9 -> 792,069 tokens) but 7,892 at 200704 (3.146e9 -> 398,658). Never extrapolate a concurrency figure across geometries. Also smoke-verified: three concurrent 67-token requests answered in parallel, ~4.1 s each - the 3 slots are real, and the pool admits 2 full-length. --- AGENTS.md | 4 +-- scripts/frank-primary.env | 14 +++++--- scripts/serve-frank-primary.sh | 5 +-- ...server-qwen38-fp8-expert-offload-tuning.md | 32 +++++++++++-------- tests/frank_primary_unit.bats | 12 ++++--- 5 files changed, 39 insertions(+), 28 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 84a5451..6632749 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (fp8 measures 7,701 B/token, so one 200704 request costs 1.546e9 B; 2.04×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) — util 0.98, and the client contract wired: `--reasoning-parser … +| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) — util 0.98, and the client contract wired: `--reasoning-parser … | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 | | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 | @@ -427,7 +427,7 @@ digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** **fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since 2026-10-02 the **shape-D** geometry: `--max-model-len 200704` × 3 slots with `--kv-cache-memory-bytes 3,150,000,000`/rank, so the pool admits **~2 -full-length conversations (2.04×)** while the argv keeps the only slot count +full-length conversations (measured: 398,658 tokens = 1.99×)** while the argv keeps the only slot count that compiles (below). **`--max-num-seqs 2` is impossible on this model.** Measured 2026-10-02 by diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index 753279d..018c5d8 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -32,11 +32,15 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" # --- qualified engine shape (2026-10-02, shape D) --------------------------- # WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots # (SEQS 3), but the POOL, not the slot count, decides how many full-length -# conversations are resident. fp8 measures 7,701 B/token (6,138 attention + -# 1,563 mamba/side; measured: 6,100,000,000 B -> 792,069 tokens), so one -# 200704-token request costs 1.546e9 B. KV_BYTES 3.15e9 admits 2.04 such -# requests; shorter requests use the spare blocks, and a third LONG request -# queues rather than being refused. +# conversations are resident. MEASURED LIVE 2026-10-02 at this shape: +# GPU KV cache size: 398,658 tokens, +# Maximum concurrency for 200,704 tokens per request: 1.99x +# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is +# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token +# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a +# concurrency figure across geometries. Shorter requests use the spare blocks, +# and a third LONG request queues rather than being refused. Verified: three +# concurrent 67-token requests answered in parallel, ~4.1 s each. # # WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes # against the then-deployed known-good shape, production stopped for each): diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh index c6c7e3a..e2c7572 100755 --- a/scripts/serve-frank-primary.sh +++ b/scripts/serve-frank-primary.sh @@ -13,8 +13,9 @@ # # Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES # 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident -# conversations at ~2 full-length (2.04x); shorter requests use the spare -# blocks. --max-num-seqs stays 3 because SEQS=2 CANNOT boot this model +# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x); +# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2 +# CANNOT boot this model # (torch.fx ConstraintViolationError on query_start_loc, measured at both # MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). NO speculative # decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index 9016515..ecffe15 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -280,7 +280,7 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap. ``` FRANK_PRIMARY_MAXLEN=200704 200704-token window FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below) -FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,701 B/token -> 2.04x) +FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x) FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized) FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env) @@ -288,12 +288,16 @@ NO --speculative-config, NO --expert-prefetch ``` **The pool, not the slot count, sets how many conversations live at once.** -fp8 costs 7,701 B/token (measured 6,138 attention + 1,563 mamba/side, -dtype-independent), so one 200704-token request costs 1.546e9 B and KV_BYTES -3.15e9 admits **2.04 full-length requests**. The endpoint advertises 3 slots: a -third LONG request queues rather than being refused, and shorter requests use -the spare blocks. Two contexts at 200704 are a capacity contract, not a -throughput one (2-way concurrency summed BELOW parity at 4064 ctx). +Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum +concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved). +The per-token price is **shape-dependent**: the 262144 shape priced at 7,701 +B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer, +so a concurrency figure never extrapolates across geometries. The endpoint +advertises 3 slots: a third LONG request queues rather than being refused, and +shorter requests use the spare blocks. Verified: three concurrent 67-token +requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a +capacity contract, not a throughput one (2-way concurrency summed BELOW parity +at 4064 ctx). #### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02) @@ -346,12 +350,12 @@ stay up: it left 0 bytes free at runtime and a real client request (995-token prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB ... 0 bytes free`. -The correction is a pool cut plus a window trim, not a downgrade. fp8 costs -7,701 B/token, so 3 x 200704 = 602,112 tokens would need 4.64e9 B for three -full-length contexts; pinning the pool at 3.15e9 instead admits 2.04 of them. -That is deliberate: the pool is the runtime-HEADROOM lever (the shape this -replaced died with 0 bytes free), while the arena is sized by `max_model_len` -alone. A third full-length request queues; short requests use the spare blocks. +The correction is a pool cut plus a window trim, not a downgrade. Three +full-length 200704-token contexts need ~4.75e9 B; pinning the pool at 3.15e9 +instead admits **1.99** of them. That is deliberate: the pool is the +runtime-HEADROOM lever (the shape this replaced died with 0 bytes free), while +the arena is sized by `max_model_len` alone. A third full-length request queues; +short requests use the spare blocks. Boot-probed 2026-10-02 (transient systemd units at the production uid + memlock, full-log capture; the rootless bench probe had masked the production @@ -364,7 +368,7 @@ memory environment). The night searched two concurrency contracts: | 3 x 237568, bf16 | PASS — KV 718,606 tokens | | 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 | | 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request | -| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 2.04x concurrency, arena 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom | +| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom | Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index 77f61b7..2be46f7 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -105,9 +105,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # max_num_reqs + 1, so the width is SEQS + 1 and MNBT only moves the # compile RANGE. # So the shape is 3 slots with a pool that admits only ~2 full-length - # conversations: fp8 measures 7,701 B/token, one 200704 request costs - # 1.546e9 B, and KV_BYTES 3.15e9 therefore gives 2.04x. Shorter requests - # use the spare blocks; a third LONG request queues. + # conversations: measured live, KV_BYTES 3.15e9/rank reports 398,658 + # tokens = 1.99x for a 200704-token request (7,892 B/token at THIS shape; + # the per-token price is shape-dependent, 7,701 at 262144). Shorter + # requests use the spare blocks; a third LONG request queues. # The arena is sized by max_model_len, NOT by KV_BYTES: the planner # reserves registry 9190 B/token x max_model_len (2.48 GiB at 262144 -> # arena 6,248 slots; 1.91 GiB at 200704 -> 6,496). The pool cut buys @@ -193,8 +194,9 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" } @test "KV dtype is fp8 — the incumbent, and the only one that boots" { - # fp8 costs 7,701 B/token, decomposed as 6,138 attention + 1,563 - # mamba/side (dtype-independent); bf16 costs 13,839 (12,276 + 1,563). + # fp8 costs ~7,701 B/token at 262144 (7,892 at 200704 — shape-dependent), + # decomposed as 6,138 attention + 1,563 mamba/side (dtype-independent); + # bf16 costs 13,839 (12,276 + 1,563). # bf16 was abandoned 2026-10-01: its 2 x 262144 boot died with 0 bytes # free on a real request, and the SEQS=2 slot count it was designed # around cannot compile this model at all (see the shape test above).