frank-primary: record the measured shape-D numbers (398,658 tokens, 1.99x, 6,496 slots)

Comments/docs only - no functional change. The live boot of c769ff2 reported: GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99x expert cache space 14.87 GiB / 6,496 slots of 2.344 MiB The arena prediction (6,496) landed exactly; the concurrency prediction (2.04x) did not, because the per-token KV price is SHAPE-DEPENDENT: 7,701 B/token at MAXLEN 262144 (6.1e9 -> 792,069 tokens) but 7,892 at 200704 (3.146e9 -> 398,658). Never extrapolate a concurrency figure across geometries. Also smoke-verified: three concurrent 67-token requests answered in parallel, ~4.1 s each - the 3 slots are real, and the pool admits 2 full-length.
2026-10-02 11:48:09 -07:00
parent c769ff2f63
commit 295d5784d3
5 changed files with 39 additions and 28 deletions
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).

 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (fp8 measures 7,701 B/token, so one 200704 request costs 1.546e9 B; 2.04×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) — util 0.98, and the client contract wired: `--reasoning-parser …
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) — util 0.98, and the client contract wired: `--reasoning-parser …
 | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
 | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |

@@ -427,7 +427,7 @@ digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
 **fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since
 2026-10-02 the **shape-D** geometry: `--max-model-len 200704` × 3 slots with
 `--kv-cache-memory-bytes 3,150,000,000`/rank, so the pool admits **~2
-full-length conversations (2.04×)** while the argv keeps the only slot count
+full-length conversations (measured: 398,658 tokens = 1.99×)** while the argv keeps the only slot count
 that compiles (below).

 **`--max-num-seqs 2` is impossible on this model.** Measured 2026-10-02 by
@@ -32,11 +32,15 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 # --- qualified engine shape (2026-10-02, shape D) ---------------------------
 # WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
 # (SEQS 3), but the POOL, not the slot count, decides how many full-length
-# conversations are resident. fp8 measures 7,701 B/token (6,138 attention +
-# 1,563 mamba/side; measured: 6,100,000,000 B -> 792,069 tokens), so one
-# 200704-token request costs 1.546e9 B. KV_BYTES 3.15e9 admits 2.04 such
-# requests; shorter requests use the spare blocks, and a third LONG request
-# queues rather than being refused.
+# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
+#   GPU KV cache size: 398,658 tokens,
+#   Maximum concurrency for 200,704 tokens per request: 1.99x
+# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is
+# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token
+# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a
+# concurrency figure across geometries. Shorter requests use the spare blocks,
+# and a third LONG request queues rather than being refused. Verified: three
+# concurrent 67-token requests answered in parallel, ~4.1 s each.
 #
 # WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
 # against the then-deployed known-good shape, production stopped for each):
@@ -13,8 +13,9 @@
 #
 # Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
 # 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
-# conversations at ~2 full-length (2.04x); shorter requests use the spare
-# blocks. --max-num-seqs stays 3 because SEQS=2 CANNOT boot this model
+# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
+# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
+# CANNOT boot this model
 # (torch.fx ConstraintViolationError on query_start_loc, measured at both
 # MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). NO speculative
 # decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
@@ -280,7 +280,7 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
 ```
 FRANK_PRIMARY_MAXLEN=200704   200704-token window
 FRANK_PRIMARY_SEQS=3          --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
-FRANK_PRIMARY_KV_BYTES=3150000000   PER RANK (fp8, 7,701 B/token -> 2.04x)
+FRANK_PRIMARY_KV_BYTES=3150000000   PER RANK (fp8, 7,892 B/token -> 1.99x)
 FRANK_PRIMARY_KV_DTYPE=fp8    --kv-cache-dtype fp8 (quantized)
 FRANK_PRIMARY_MNBT=1024       --max-num-batched-tokens 1024
 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
@@ -288,12 +288,16 @@ NO --speculative-config, NO --expert-prefetch
 ```

 **The pool, not the slot count, sets how many conversations live at once.**
-fp8 costs 7,701 B/token (measured 6,138 attention + 1,563 mamba/side,
-dtype-independent), so one 200704-token request costs 1.546e9 B and KV_BYTES
-3.15e9 admits **2.04 full-length requests**. The endpoint advertises 3 slots: a
-third LONG request queues rather than being refused, and shorter requests use
-the spare blocks. Two contexts at 200704 are a capacity contract, not a
-throughput one (2-way concurrency summed BELOW parity at 4064 ctx).
+Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
+concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
+The per-token price is **shape-dependent**: the 262144 shape priced at 7,701
+B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer,
+so a concurrency figure never extrapolates across geometries. The endpoint
+advertises 3 slots: a third LONG request queues rather than being refused, and
+shorter requests use the spare blocks. Verified: three concurrent 67-token
+requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a
+capacity contract, not a throughput one (2-way concurrency summed BELOW parity
+at 4064 ctx).

 #### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02)

@@ -346,12 +350,12 @@ stay up: it left 0 bytes free at runtime and a real client request (995-token
 prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
 ... 0 bytes free`.

-The correction is a pool cut plus a window trim, not a downgrade. fp8 costs
-7,701 B/token, so 3 x 200704 = 602,112 tokens would need 4.64e9 B for three
-full-length contexts; pinning the pool at 3.15e9 instead admits 2.04 of them.
-That is deliberate: the pool is the runtime-HEADROOM lever (the shape this
-replaced died with 0 bytes free), while the arena is sized by `max_model_len`
-alone. A third full-length request queues; short requests use the spare blocks.
+The correction is a pool cut plus a window trim, not a downgrade. Three
+full-length 200704-token contexts need ~4.75e9 B; pinning the pool at 3.15e9
+instead admits **1.99** of them. That is deliberate: the pool is the
+runtime-HEADROOM lever (the shape this replaced died with 0 bytes free), while
+the arena is sized by `max_model_len` alone. A third full-length request queues;
+short requests use the spare blocks.

 Boot-probed 2026-10-02 (transient systemd units at the production uid +
 memlock, full-log capture; the rootless bench probe had masked the production
@@ -364,7 +368,7 @@ memory environment). The night searched two concurrency contracts:
 | 3 x 237568, bf16 | PASS — KV 718,606 tokens |
 | 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
 | 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
-| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 2.04x concurrency, arena 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
+| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |

 Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
 footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
@@ -105,9 +105,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
    # max_num_reqs + 1, so the width is SEQS + 1 and MNBT only moves the
    # compile RANGE.
    # So the shape is 3 slots with a pool that admits only ~2 full-length
-    # conversations: fp8 measures 7,701 B/token, one 200704 request costs
-    # 1.546e9 B, and KV_BYTES 3.15e9 therefore gives 2.04x. Shorter requests
-    # use the spare blocks; a third LONG request queues.
+    # conversations: measured live, KV_BYTES 3.15e9/rank reports 398,658
+    # tokens = 1.99x for a 200704-token request (7,892 B/token at THIS shape;
+    # the per-token price is shape-dependent, 7,701 at 262144). Shorter
+    # requests use the spare blocks; a third LONG request queues.
    # The arena is sized by max_model_len, NOT by KV_BYTES: the planner
    # reserves registry 9190 B/token x max_model_len (2.48 GiB at 262144 ->
    # arena 6,248 slots; 1.91 GiB at 200704 -> 6,496). The pool cut buys
@@ -193,8 +194,9 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
 }

@test "KV dtype is fp8 — the incumbent, and the only one that boots" {
-    # fp8 costs 7,701 B/token, decomposed as 6,138 attention + 1,563
-    # mamba/side (dtype-independent); bf16 costs 13,839 (12,276 + 1,563).
+    # fp8 costs ~7,701 B/token at 262144 (7,892 at 200704 — shape-dependent),
+    # decomposed as 6,138 attention + 1,563 mamba/side (dtype-independent);
+    # bf16 costs 13,839 (12,276 + 1,563).
    # bf16 was abandoned 2026-10-01: its 2 x 262144 boot died with 0 bytes
    # free on a real request, and the SEQS=2 slot count it was designed
    # around cannot compile this model at all (see the shape test above).