frank-primary: 3 slots x 200704, fp8, pool sized for 2 contexts

Re-applies the desired shape incrementally on the proven-good fp8 3x262144
base, after bisecting why the first attempt (30904ba) crash-looped.

Root cause: `--max-num-seqs 2` cannot compile on this model. AOT
build_guards(strict_error=True) inside profile_run -> _dummy_run specializes
`query_start_loc`'s width to SEQS + 1 (3 at SEQS 2, 4 at SEQS 3) and the
ConstraintViolationError kills engine init ~7 min in. MNBT is orthogonal:
SEQS 2 failed at MNBT 1024 AND 2048; SEQS 3 passed at both. KV dtype was a
red herring. Measured by four single-variable boot probes.

Arena lever measured: the expert arena is sized from `max_model_len` via the
registry-priced KV reserve (9190 B/token x max_model_len), NOT from
--kv-cache-memory-bytes. 262144 -> 14.30 GiB / 6,248 slots; 200704 ->
14.87 GiB / 6,496 slots. The 6.1e9 -> 3.15e9 pool cut bought zero arena and
is a runtime-headroom lever.

Shape: MAXLEN 200704, SEQS 3 (unchanged - the only slot count that compiles),
KV_BYTES 3.15e9/rank = 2.04x full-length concurrency at 7,701 B/token fp8,
MNBT 1024, gather 32x64, no spec decode, no prefetch, no precise-memory.

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **2 seqs × 200704 ctx** at KV_BYTES 3,150,000,000/rank (fp8 measures 7,701 B/token), pool deliberately small so the expert arena keeps what it reclaims (~7,500 slots vs 6,248 at the 6.1e9 shape) — util 0.98, and the client contract wired: `--reasoning-parser …
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (fp8 measures 7,701 B/token, so one 200704 request costs 1.546e9 B; 2.04×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) — util 0.98, and the client contract wired: `--reasoning-parser …
 | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
 | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
 
@@ -419,30 +419,49 @@ Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
 (venv on disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was
 staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02).
 
-### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
+### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
 
 The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
 digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
 (~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
-**MTP4** speculative decoding, **expert prefetch OFF**, 200,000 ctx x 3 seqs,
+**fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since
-**fp8 KV**. Deployed 2026-10-01 by rollback from commit `2339fae` after the
+2026-10-02 the **shape-D** geometry: `--max-model-len 200704` × 3 slots with
-GLM/DFlash2 experiment was declared failed (same day).
+`--kv-cache-memory-bytes 3,150,000,000`/rank, so the pool admits **~2
+full-length conversations (2.04×)** while the argv keeps the only slot count
+that compiles (below).
 
-Measured 2026-09-29 tuning campaign: **decode 37.4-40.6 tok/s, acceptance
+**`--max-num-seqs 2` is impossible on this model.** Measured 2026-10-02 by
-~54 %** (MTP4 + prefetch off — the fleet's only endpoint that meets the
+single-variable boot probe against the then-deployed shape: SEQS 2 fails at
-35 tok/s target). 256k ctx arm measured 34.1-36.5; 2-way concurrency sums
+BOTH MNBT 1024 and 2048 with `torch.fx.experimental.symbolic_shapes.Constraint
-BELOW parity (25.49 vs 26.8 single) — slots are a capacity feature, not a
+ViolationError: Constraints violated (L['query_start_loc'].size()[0]) …
-throughput one. V-only q8 KV is NOT possible via config: stock
+specialized it to be a constant (3)`; SEQS 3 passes at both. `model_runner.py`
-`--kv-cache-dtype` is K+V together; a V-only quant needs a fork patch.
+sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
+`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
+`SEQS + 1` for any `MNBT ≥ SEQS` — MNBT only moves the compile RANGE
+((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
+graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.
 
-The engine auto-sets the attention block size (mamba-page floor); the
+**The expert arena's lever is `max_model_len`, not `--kv-cache-memory-bytes`.**
-expert plan's registry prices KV per token; the expert arena (6,574 slots
+The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
-x 2.344 MiB = 15.05 GiB/rank) is the decode-critical region — cutting it
+graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
-for KV headroom collapses the offload hit rate.
+`max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
+200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena
+0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime
+headroom (the shape it replaced died with `0 bytes free` on a real 995-token
+request). Grow the arena by lowering `max_model_len`. The arena (2.344 MiB per
+slot) is the decode-critical region — cutting it for KV headroom collapses the
+offload hit rate.
+
+Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode
+**37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5;
+2-way concurrency sums BELOW parity (25.49 vs 26.8 single) — slots are a
+capacity feature, not a throughput one. V-only q8 KV is NOT possible via
+config: stock `--kv-cache-dtype` is K+V together; a V-only quant needs a fork
+patch.
 
 Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
 `User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
-frank-primary-glm`. The serve script drives `podman run` directly, gates
+frank-primary-qwen`. The serve script drives `podman run` directly, gates
 the digest-pinned image, the payload's `config.json` and its exact
 131-shard count and the GPU device nodes, clears stale container
 identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
@@ -849,7 +868,7 @@ Active:
 |---------|---------|
 | frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md` |
 | frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` |
-| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
+| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
 

scripts/frank-primary.env

@@ -1,40 +1,10 @@
 # frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
 #
-# 2026-10-01 cutover to the BF16-KV no-spec incumbent (the overnight redo
+# 2026-10-02 shape D: fp8 KV, MAXLEN 200704 x SEQS 3, the KV pool cut to ~2
-# campaign's verdict, Frank's call): same digest-pinned engine and payload as
+# full-length contexts. The window shrinks from the native 262144 to 200704 and
-# the 2026-09-29 cutover, but the KV cache is UNQUANTIZED (auto = bf16) and
+# the pool from 6.1e9 to 3.15e9; --max-num-seqs stays 3 because SEQS=2 CANNOT
-# speculative decode is GONE. Git tag frank-primary-fp8-mtp-20261001 is the
+# boot this model (measured 2x2 below). Same digest-pinned tcclaviger RDNA4
-# last pre-cutover commit; the old shape (fp8 KV + MTP4) is in history and
+# fork. Rollback for the fp8+MTP4 era is git tag frank-primary-fp8-mtp-20261001.
-# restorable from the tag.
-#
-# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
-#   * MAXLEN 241664 x SEQS 3 (724,992 tokens) at bf16 KV is the DEPLOYED
-#     shape (2026-10-01 night ladder): KV_BYTES 10,184,000,000 PER RANK,
-#     engine measured "KV cache size: 732,359 tokens; Maximum concurrency
-#     for 241,664 tokens per request: 3.03x". 262144 and 245760 both OOMed
-#     at graph capture (76 MiB short / 0 bytes free); 237568 passed at
-#     718,606 tokens and the +4096 bonus step passed and won.
-#   * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
-#     the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
-#     the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
-#     11.05 GiB/rank and boots clean — the ceiling was spec's, not VRAM's.
-#   * bf16 KV doubles the attention-KV bytes/token (measured 14,047 vs
-#     7,926) and costs ~14% decode vs fp8 at 4064 ctx (49.25 -> 36.42 with
-#     the gather win) — traded deliberately for clean cache + 262k windows.
-#   * GATHER 32x64 is the night's biggest lever: R4D_LRU_GATHER_CHUNKS=32 /
-#     R4D_LRU_GATHER_LANES=64 measured +18.7% decode vs default geometry,
-#     and the win grows with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 fp8
-#     era). Do not drop it.
-#   * MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s
-#     vs 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5%
-#     and arena. 1024 keeps the workspace floor so the arena stays max.
-#   * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
-#     prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
-#     roughly half the gather traffic was waste. Never re-add
-#     --expert-prefetch without a fresh A/B.
-#   * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
-#     offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
-#     numerics.
 #
 # HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
 # prefixed FRANK_PRIMARY_ so none can leak.
@@ -59,43 +29,63 @@ FRANK_PRIMARY_PUBLISH=7770:7770
 # hoc experiment identity (both hold the GPUs / name registrations).
 FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 
-# --- qualified engine shape (2026-10-01, RESTORED KNOWN-GOOD: fp8 3 windows) ---
+# --- qualified engine shape (2026-10-02, shape D) ---------------------------
-# bf16 at 2 x 262144 did NOT stay up: with and without --expert-precise-memory
+# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
-# the boot failed (`Engine core initialization failed`, NRestarts climbing).
+# (SEQS 3), but the POOL, not the slot count, decides how many full-length
-# The last shape measured serving was fp8 3 x 262144 / MNBT 2048 / KV 6.1e9
+# conversations are resident. fp8 measures 7,701 B/token (6,138 attention +
-# (KV 792,069 tokens, 3.02x, arena 6,248, ~3.6 GiB free) and is restored here
+# 1,563 mamba/side; measured: 6,100,000,000 B -> 792,069 tokens), so one
-# while the bf16 path is investigated in a window (see log.md 2026-10-01).
+# 200704-token request costs 1.546e9 B. KV_BYTES 3.15e9 admits 2.04 such
-# Candidate next test: bf16 with SEQS=3 to isolate the KV dtype from the
+# requests; shorter requests use the spare blocks, and a third LONG request
-# sequence count, and/or --enforce-eager to bypass the dynamo constraint path.
+# queues rather than being refused.
-# SEQS 2 x MAXLEN 262144 at BF16 KV (Frank's call, 2026-10-01): two contexts of
+#
-# the model's FULL native window with UNQUANTIZED KV, plus
+# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
-# --expert-precise-memory to hand the arena the compile/graph-capture headroom.
+# against the then-deployed known-good shape, production stopped for each):
-# Pool: bf16 costs 13,839 B/token (measured: 11,047,010,304 B -> 798,280 tokens
+#   SEQS 2 + MNBT 1024 -> Engine core initialization failed
-# on the 3x241664 bf16 boot; decomposes as 12,276 attention + 1,563 mamba/side
+#   SEQS 2 + MNBT 2048 -> Engine core initialization failed
-# caches, against fp8's 6,138 + 1,563 = 7,701). 2 x 262144 = 524,288 tokens
+#   SEQS 3 + MNBT 1024 -> healthy (792,069 tokens, 3.02x, arena 6,248)
-# -> 7.256e9 B floor, pinned at 7,300,000,000 (~527,500 tokens, 3,206 margin).
+#   SEQS 3 + MNBT 2048 -> healthy (the deployed incumbent)
-# Budget (per rank): 31.22 GiB total; non-expert weights 6.28, graphs 0.55,
+# The failure is torch.fx ConstraintViolationError: query_start_loc.size()[0]
-# non-torch ~1.45, pool 6.76 -> ~29.35 used, ~2.5 GiB free (measured basis:
+# "marked dynamic but your code specialized it to be a constant (3)", raised
-# the fp8 3-window shape used 28.27 and left 3.59).
+# from AOT build_guards(strict_error=True) during profile_run -> _dummy_run.
-# PRECISE MEMORY: DROPPED 2026-10-01 after it crashed the boot in combination
+# model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets
-# with bf16. The flag selects the fitted 5.25 GiB overhead instead of the
+# num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is
-# 7.60 GiB default, moving the plan's `room` and therefore the arena
+# SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT only moves the
-# (6,248 -> ~7,278 slots). But the arena is DEVICE-resident, and the budget
+# compile RANGE ((1,1024) vs (1,2048)) and cannot dodge it — the 2x2 above is
-# for this shape does not have 2.35 GiB to spare:
+# why MNBT is not a lever here. --enforce-eager bypasses the path but loses
-#   weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena 16.65
+# CUDA graphs. Same failure class killed the 2026-10-01 bf16 2 x 262144 shape.
-#   + non-torch ~1.45 = 31.73 of 31.86 GiB  =>  ~0.13 GiB free at boot
+#
-# measured as a CrashLoop: every attempt reached graph capture, the engine
+# THE ARENA LEVER IS max_model_len, NOT KV_BYTES (measured 2026-10-02). The
-# exited cleanly, systemd restarted it (NRestarts=3 at ~6.7 min each).
+# planner sizes expert cache as budget - non-expert - KV reserve - graphs -
-# With the default overhead the arena is 14.31 GiB and the same shape totals
+# overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len:
-# ~29.4 GiB, leaving ~2.4 GiB for capture. Re-enable the flag only with a
+#   MAXLEN 262144 -> reserve 2.48 GiB -> arena 14.30 GiB / 6,248 slots
-# pool/context reduction that pays for it.
+#   MAXLEN 200704 -> reserve 1.91 GiB -> arena 14.87 GiB / 6,496 slots
-# MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so
+# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
-# 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096.
+# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
-FRANK_PRIMARY_MAXLEN=262144
+# died with 0 bytes free on a real 995-token request). To grow the arena,
+# lower MAXLEN — not KV_BYTES.
+#
+# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
+# measured +18.7% decode vs default geometry and the win grows with depth.
+# Do not drop it.
+# MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs
+# 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5% and
+# arena. 1024 keeps the workspace floor so the arena stays max. (Side note
+# measured 2026-10-02: MNBT 2048 compiles the first graph in 26 s vs 172 s at
+# 1024 — a startup-cost lever, not a correctness one.)
+# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
+# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
+# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
+# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
+# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
+# --expert-precise-memory stays DROPPED: it crashed the 2 x 262144 bf16 boot
+# (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each). It survives in
+# the serve script as an opt-in; re-enable only with a pool/context reduction
+# that pays for the 2.35 GiB it moves into the arena.
+FRANK_PRIMARY_MAXLEN=200704
 FRANK_PRIMARY_SEQS=3
-FRANK_PRIMARY_KV_BYTES=6100000000
+FRANK_PRIMARY_KV_BYTES=3150000000
 FRANK_PRIMARY_KV_DTYPE=fp8
-FRANK_PRIMARY_MNBT=2048
+FRANK_PRIMARY_MNBT=1024
 FRANK_PRIMARY_GATHER_CHUNKS=32
 FRANK_PRIMARY_GATHER_LANES=64
 FRANK_PRIMARY_UTIL=0.98
@@ -114,4 +104,4 @@ FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
 # preserve_thinking keeps prior thinking blocks in the rendered prompt across
 # turns; enable_thinking makes thinking the server-side default. Verified the
 # payload's tokenizer_config template consumes both keys.
-FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
+FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'

scripts/frank-primary.service

@@ -1,8 +1,9 @@
 # frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
-# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
+# Qwen3.8-Flash-Next-Uncensored-FP8, fp8 KV, NO spec decode, 3 slots x 200704
-# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
+# context with the pool sized to ~2 full-length conversations, TP=2, tiered
-# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
+# experts, 131-shard payload) on the 2x R9700 pair, dankserver :7770.
-# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
+# 2026-09-29 cutover from the R9V radiance fork (git tag
+# frank-primary-r9v-q8-20260929 is the pre-cutover state; its
 # launcher/shim/setup-state machinery is retired and the state dirs stay on
 # disk as rollback).
 #

scripts/serve-frank-primary.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 # serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
-# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
+# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700,
 # dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
 # frank-primary-r9v-q8-20260929 is the pre-cutover state).
 #
@@ -11,14 +11,17 @@
 # every flag here was qualified by boot probe on 2026-09-29 — see
 # frank-primary.env for the qualification record).
 #
-# Shape (2026-10-01 bf16 cutover): MAXLEN 262144 x 3 sessions (786,432
+# Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
-# tokens), KV UNQUANTIZED (--kv-cache-dtype auto = bf16; the fp8 q8/q8 shape
+# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
-# is retired to git tag frank-primary-fp8-mtp-20261001), KV_BYTES
+# conversations at ~2 full-length (2.04x); shorter requests use the spare
-# 11,047,010,304 PER RANK (the engine measured 798,280 tokens of capacity /
+# blocks. --max-num-seqs stays 3 because SEQS=2 CANNOT boot this model
-# 3.05x concurrency), NO speculative decode, --max-num-batched-tokens 1024
+# (torch.fx ConstraintViolationError on query_start_loc, measured at both
-# (Frank's lock), MoE gather geometry R4D_LRU_GATHER_CHUNKS=32 /
+# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). NO speculative
-# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix caching,
+# decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
-# expert prefetch OFF (36% precision measured), TunableOp off.
+# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
+# with depth), prefix caching, expert prefetch OFF (36% precision measured),
+# TunableOp off. The native-262144 and fp8+MTP4 shapes are retired to git tag
+# frank-primary-fp8-mtp-20261001.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
 # LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host

serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

@@ -275,17 +275,67 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
 behavior (the gather win growing with depth, bf16's decode cost at depth)
 is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
 
-### The deployed shape (2026-10-02: 2 x 200704, fp8, pool cut for the arena)
+### The deployed shape (2026-10-02: 3 slots x 200704, fp8, pool sized for 2 contexts)
 
 ```
-FRANK_PRIMARY_MAXLEN=200704    x SEQS 2 (401,408 tokens of contract)
+FRANK_PRIMARY_MAXLEN=200704   200704-token window
-FRANK_PRIMARY_KV_BYTES=3150000000   PER RANK (fp8, 7,701 B/token)
+FRANK_PRIMARY_SEQS=3          --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
-FRANK_PRIMARY_KV_DTYPE=fp8   --kv-cache-dtype fp8 (quantized)
+FRANK_PRIMARY_KV_BYTES=3150000000   PER RANK (fp8, 7,701 B/token -> 2.04x)
+FRANK_PRIMARY_KV_DTYPE=fp8    --kv-cache-dtype fp8 (quantized)
 FRANK_PRIMARY_MNBT=1024       --max-num-batched-tokens 1024
 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
 NO --speculative-config, NO --expert-prefetch
 ```
 
+**The pool, not the slot count, sets how many conversations live at once.**
+fp8 costs 7,701 B/token (measured 6,138 attention + 1,563 mamba/side,
+dtype-independent), so one 200704-token request costs 1.546e9 B and KV_BYTES
+3.15e9 admits **2.04 full-length requests**. The endpoint advertises 3 slots: a
+third LONG request queues rather than being refused, and shorter requests use
+the spare blocks. Two contexts at 200704 are a capacity contract, not a
+throughput one (2-way concurrency summed BELOW parity at 4064 ctx).
+
+#### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02)
+
+Four single-variable boot probes against the then-deployed shape, production
+stopped for each:
+
+| MAXLEN | SEQS | MNBT | result |
+|---|---|---|---|
+| 262144 | 2 | 1024 | **Engine core initialization failed** |
+| 262144 | 2 | 2048 | **Engine core initialization failed** |
+| 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
+| 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
+
+The failure is `torch.fx.experimental.symbolic_shapes.ConstraintViolationError:
+Constraints violated (L['query_start_loc'].size()[0]) ... specialized it to be
+a constant (3)`, raised from AOT `build_guards(strict_error=True)` during
+`determine_available_memory -> profile_run -> _dummy_run`. `model_runner.py`
+sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
+`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
+`SEQS + 1` for ANY `MNBT >= SEQS`: 3 at SEQS 2, 4 at SEQS 3. **MNBT is not a
+lever here** - it only moves the compile RANGE ((1,1024) vs (1,2048)), which
+is why the identical error appears at both rungs. `--enforce-eager` bypasses
+the path at the cost of CUDA graphs. The same failure class killed the
+2026-10-01 bf16 2 x 262144 shape.
+
+#### The arena lever is `max_model_len`, not `--kv-cache-memory-bytes`
+
+The fork's planner sizes expert cache as `budget - non-expert - KV reserve -
+graphs - overhead`, and the KV reserve is registry-priced at **9190 B/token x
+max_model_len** - not from the `--kv-cache-memory-bytes` value:
+
+| MAXLEN | KV reserve | arena |
+|---|---|---|
+| 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
+| 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
+
+D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
+pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
+which is still worth having - the shape it replaced died with `0 bytes free`
+on a real 995-token request. Grow the arena by lowering `max_model_len`; the
+pool is the headroom lever.
+
 This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
 `--expert-precise-memory` — which crash-looped at graph capture: the arena is
 DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
@@ -296,14 +346,12 @@ stay up: it left 0 bytes free at runtime and a real client request (995-token
 prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
 ... 0 bytes free`.
 
-The correction is a pool cut, not a downgrade. fp8 costs 7,701 B/token
+The correction is a pool cut plus a window trim, not a downgrade. fp8 costs
-(measured 6,138 attention + 1,563 mamba/side, dtype-independent), so 2 x
+7,701 B/token, so 3 x 200704 = 602,112 tokens would need 4.64e9 B for three
-200704 = 401,408 tokens needs 3.091e9 B; pinned at 3.15e9, ~44 MB of margin
+full-length contexts; pinning the pool at 3.15e9 instead admits 2.04 of them.
-above the floor — the same convention the 6.1e9 shape used. The arena is what
+That is deliberate: the pool is the runtime-HEADROOM lever (the shape this
-remains: the planner sizes it AFTER the KV override, so the 6,248 slots of the
+replaced died with 0 bytes free), while the arena is sized by `max_model_len`
-6.1e9 shape become roughly 7,500 here. Two contexts at 200704 are a capacity
+alone. A third full-length request queues; short requests use the spare blocks.
-contract, not a throughput one (2-way concurrency summed BELOW parity at 4064
-ctx), so the slot buys the VRAM headroom long generations need.
 
 Boot-probed 2026-10-02 (transient systemd units at the production uid +
 memlock, full-log capture; the rootless bench probe had masked the production
@@ -316,11 +364,11 @@ memory environment). The night searched two concurrency contracts:
 | 3 x 237568, bf16 | PASS — KV 718,606 tokens |
 | 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
 | 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
-| **2 x 200704, fp8, pool cut for the arena** | **DEPLOYED (2026-10-02, Frank's correction)**: fp8 KV, two 200704 windows, ~7,500 arena slots, ~3.6 GiB free VRAM measured |
+| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 2.04x concurrency, arena 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
-7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
+
-decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
+Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
-graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144
+footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
-needs 12.47 GB even at fp8.
+at fp8.
 
 The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
 reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
@@ -379,8 +427,8 @@ Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2
 No such file or directory` — measured 2026-10-01, which silently cost a whole
 arm's numbers. A window script must do the copy, not assume the path exists.
 
-### The deployed shape, measured again (2026-10-01 A/B window)
+### The then-deployed shape, measured again (2026-10-01 A/B window)
-A clean production-environment boot of the deployed config (4 x 262144 fp8, MNBT
+A clean production-environment boot of the config deployed at that time (4 x 262144 fp8, MNBT
 2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
 concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
 deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is

tests/frank_primary_unit.bats

@@ -90,28 +90,35 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
 }
 
 @test "the qualified engine shape is in the env file" {
-    # 2026-10-01 night ladder (Frank's method), then a runtime-survivability
+    # 2026-10-02 shape D. History: the 2026-10-01 night ladder put SEQS=3
-    # correction. SEQS=3 bf16 topped out at 241664 (KV 732,359 / 3.03x);
+    # bf16 at 241664 (KV 732,359 / 3.03x); SEQS=6 fp8 does NOT fit (6 x
-    # SEQS=6 fp8 does NOT fit (6 x 241664 asked 10.70 GiB against 9.18 GiB
+    # 241664 asked 10.70 GiB against 9.18 GiB free); SEQS=5 fp8 x 262144
-    # free); SEQS=5 fp8 x 262144 BOOTED (KV 1,350,182 / 5.15x) but left ~0
+    # BOOTED (1,350,182 / 5.15x) but left ~0 bytes free and a real 995-token
-    # bytes free at runtime and a real client request (995-token prompt with
+    # request killed the worker with `Tried to allocate 50.00 MiB ... 0 bytes
-    # max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
+    # free`. UTIL stays 0.98 (lowering it shrinks the engine's own budget
-    # ... 0 bytes free`. FOUR slots at the full native 262144 window is the
+    # instead of buying slack).
-    # deployed shape: ~1.93 GiB more headroom per rank for
+    # SEQS=2 is IMPOSSIBLE on this model, measured 2026-10-02 by
-    # activations/graphs. UTIL stays 0.98 (lowering it shrinks the engine's
+    # single-variable boot probe against the then-deployed shape: SEQS 2 fails
-    # own budget instead of buying slack).
+    # at BOTH MNBT 1024 and 2048 with torch.fx ConstraintViolationError
-    # FINAL (2026-10-01, Frank): TWO full-size windows with BF16 KV plus
+    # (query_start_loc.size()[0] marked dynamic but specialized to a constant
-    # --expert-precise-memory. bf16 costs 13,839 B/token (measured), so
+    # (3)); SEQS 3 passes at both. model_runner sizes that buffer
-    # 2 x 262144 = 524,288 tokens needs 7.256e9 B; pinned at 7.3e9 (~527.5k
+    # max_num_reqs + 1, so the width is SEQS + 1 and MNBT only moves the
-    # tokens, 3,206 margin). The precise flag drops the plan's overhead
+    # compile RANGE.
-    # 7.60 -> 5.25 GiB, moving the arena from 6,248 to ~7,278 slots.
+    # So the shape is 3 slots with a pool that admits only ~2 full-length
-    grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
+    # conversations: fp8 measures 7,701 B/token, one 200704 request costs
+    # 1.546e9 B, and KV_BYTES 3.15e9 therefore gives 2.04x. Shorter requests
+    # use the spare blocks; a third LONG request queues.
+    # The arena is sized by max_model_len, NOT by KV_BYTES: the planner
+    # reserves registry 9190 B/token x max_model_len (2.48 GiB at 262144 ->
+    # arena 6,248 slots; 1.91 GiB at 200704 -> 6,496). The pool cut buys
+    # runtime headroom, not arena.
+    grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_KV_BYTES=6100000000$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
-    grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
+    grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
-    # --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the boot
+    # --expert-precise-memory is NOT set: it put the bf16 2 x 262144 boot
     # 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and
     # crash-looped at graph capture. The serve script keeps the opt-in flag.
     ! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
@@ -185,14 +192,12 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     ! grep -q -- '--speculative-config' "$SERVE"
 }
 
-@test "KV dtype is bf16 (auto) — exactness, paid for in pool bytes" {
+@test "KV dtype is fp8 — the incumbent, and the only one that boots" {
-    # --kv-cache-dtype auto = bf16 (unquantized). Frank's call 2026-10-01.
+    # fp8 costs 7,701 B/token, decomposed as 6,138 attention + 1,563
-    # bf16 costs 13,839 B/token against fp8's 7,701 — decomposed as 12,276
+    # mamba/side (dtype-independent); bf16 costs 13,839 (12,276 + 1,563).
-    # attention + 1,563 mamba/side (dtype-independent) versus fp8's 6,138 +
+    # bf16 was abandoned 2026-10-01: its 2 x 262144 boot died with 0 bytes
-    # 1,563. That halves the pool capacity per byte, which is why the contract
+    # free on a real request, and the SEQS=2 slot count it was designed
-    # is two full-size windows instead of three; in exchange the KV is exact
+    # around cannot compile this model at all (see the shape test above).
-    # (no quantization error) and measured +4% decode at shallow depth, where
-    # fp8's dequant cost dominates over the extra bytes read.
     grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
     grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
 }
@@ -208,13 +213,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
 }
 
-@test "MNBT is at Frank's 2048 with the arena intact" {
+@test "MNBT is at Frank's 1024 with the arena intact" {
     # MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
     # interchangeable on decode AND identical on arena (the sweep measured
     # 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena.
-    # 2048 is Frank's setting as of 2026-10-01 — it keeps the workspace floor
+    # 1024 is Frank's lock — it keeps the workspace floor so the expert arena
-    # so the expert arena stays max.
+    # stays max. (2026-10-02: MNBT 2048 compiles the first graph in 26 s vs
-    grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
+    # 172 s at 1024 — a startup-cost lever, not a correctness one.)
+    grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
     grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
 }