frank-small: reslice to 2 x 262,144 native slots (was 4 x 151,552)

The KV pool stays pinned at 606,208 tokens (card is full: 531 MiB free at
the old shape; autofit's 687,388 would leave 0.53 GB and OOM on the first
long request). 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack.

Motivation: harness compaction. frank-primary's contextWindow 262,144
compacts at 70% = 183,501 tokens, and the compaction request carries a
32,768-token summary budget -> 216,269 must fit ONE frank-small request.
The 151,552 slot window could not hold it; the native window does.

No spec-decode interaction: checkpoint ships zero mtp.* tensors (config
lies), seam keeps every --speculative-* flag away from the engine.

AGENTS.md

@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 |--------|---------|---------|-------|------|
 | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
 | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
-| Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 4 slots of 151,552, bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 |
+| Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 2 slots of 262,144 (native window; 2026-10-01 reslice from 4 x 151,552 for compaction headroom), bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 |
 
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -414,10 +414,14 @@ dtype* (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
 pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` (795 MiB).
 The KV pool is then **pinned at 606,208 tokens** (`--max-total-tokens`) —
 the engine's autofit picks 687,388 and leaves 0.53 GB, which Triton's
-deferred kernel loads eat and the first long request OOMs on. Four
+deferred kernel loads eat and the first long request OOMs on. Two
-151,552-token slots; a single request may grow to the whole pool (170k
+262,144-token slots at the model's NATIVE window (2026-10-01 reslice from
-verified). Prefill runs **eager** (`--disable-prefill-cuda-graph`) — there
+4 x 151,552: 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack, and
-is no headroom for capture; decode graphs are on.
+a compaction request — frank-primary's trigger is 183,501 tokens plus a
+32,768-token summary budget = 216,269 — now fits ONE request); a single
+request may grow to the whole pool. Prefill runs **eager**
+(`--disable-prefill-cuda-graph`) — there is no headroom for capture;
+decode graphs are on.
 
 **No speculative decode — a checkpoint fact.** `text_config` declares
 `mtp_num_hidden_layers: 1`, but the shipped safetensors (base BF16 and this

scripts/frank-small.env

@@ -28,8 +28,10 @@
 # pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
 # take most of it after boot, and eager-prefill activations then OOM on
 # the first long request. The pool is therefore PINNED at 606,208 tokens
-# (4 x 151,552), which leaves ~0.8 GB for the prefill workspace; a 170k
+# (2 x 262,144 native slots since the 2026-10-01 reslice; 4 x 151,552
-# single request and 4 concurrent 54k requests are verified against it.
+# before), which leaves ~0.8 GB for the prefill workspace; a 170k single
+# request and 4 concurrent 54k requests were verified at the 4-slot
+# shape.
 # The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
 #
 # Sampling: greedy by default (the fleet contract); a client that sends a
@@ -54,14 +56,17 @@ ALIAS=frank-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771
 
-# Four slots on fp8 KV. The pool is pinned (see SIZING above): 606,208
+# Two slots at the model's native 262,144 window on fp8 KV. The pool is
-# tokens = 4 x 151,552, so the slot window is 151,552. A single request
+# pinned (see SIZING above): 606,208 tokens, and 2 x 262,144 = 524,288
-# may still grow to the whole pool (a 170k prompt is verified); the grid
+# leaves ~82k tokens of pool slack. Resliced 2026-10-01 from 4 x 151,552
-# is the four-way concurrency contract.
+# so a compaction request (frank-primary's trigger 183,501 tokens + a
+# 32,768-token summary budget = 216,269) fits ONE request; the grid is
+# the two-way concurrency contract. A single request may still grow to
+# the whole pool (a 170k prompt was verified at the 4-slot shape).
 MEM_FRACTION=0.98
 MAX_TOTAL_TOKENS=606208
-CONTEXT_LENGTH=151552
+CONTEXT_LENGTH=262144
-MAX_RUNNING=4
+MAX_RUNNING=2
 KV_DTYPE=fp8_e4m3
 QUANTIZATION=auto-round
 # DeltaNet (mamba) recurrent states: the ssm state pool is the single
@@ -70,8 +75,9 @@ QUANTIZATION=auto-round
 # slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
 # bf16 dtype + 32 slots = 795 MiB.
 MAMBA_SSM_DTYPE=bfloat16
-# 32 slots is 8x the 4-way concurrency: enough for the radix mamba cache
+# 32 slots is 16x the 2-way concurrency: enough for the radix mamba
-# to keep shared-prefix states, small enough to leave the KV pool room.
+# cache to keep shared-prefix states, small enough to leave the KV pool
+# room.
 MAX_MAMBA_CACHE_SIZE=32
 
 # Reasoning is extracted into reasoning_content; tools use the qwen3_coder

scripts/serve-frank-small.sh

@@ -14,7 +14,7 @@
 # WITHOUT speculative decoding — a NEXTN draft would be an uninitialised
 # 4.24 GiB ghost model drafting at 0% acceptance.
 #
-# Context: CONTEXT_LENGTH x MAX_RUNNING=4 slots must fit the fp8 KV pool,
+# Context: CONTEXT_LENGTH x MAX_RUNNING slots must fit the fp8 KV pool,
 # which the engine autofits around the weights AND the DeltaNet ssm state
 # pool (see frank-small.env). The mamba pool defaults to ~5.1 GB, so it is
 # pinned (bf16, 32 slots). With the grid at its measured fit the card has

serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md

@@ -39,11 +39,15 @@ timestamp: 2026-10-01
   (4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
   DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
   tower (27 layers) in the safetensors — no separate mmproj file.
-- **Context:** four 151,552-token slots (`--context-length 151552`,
+- **Context:** two 262,144-token slots at the model's native window
-  `--max-running-requests 4`) against a **pinned** 606,208-token pool
+  (`--context-length 262144`, `--max-running-requests 2`) against a
-  (`--max-total-tokens`). A single request may grow to the whole pool —
+  **pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
-  a 170,019-token prompt is verified — the grid is the four-way
+  reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
-  concurrency contract.
+  ~82k tokens of pool slack, and a compaction request (frank-primary's
+  trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
+  one request. A single request may grow to the whole pool — a
+  170,019-token prompt was verified at the 4-slot shape — the grid is
+  the two-way concurrency contract.
 - **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
   dim x 1 B), pool 9.31 GiB.
 - **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet