frank-small: reslice to 2 x 262,144 native slots (was 4 x 151,552)

The KV pool stays pinned at 606,208 tokens (card is full: 531 MiB free at
the old shape; autofit's 687,388 would leave 0.53 GB and OOM on the first
long request). 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack.

Motivation: harness compaction. frank-primary's contextWindow 262,144
compacts at 70% = 183,501 tokens, and the compaction request carries a
32,768-token summary budget -> 216,269 must fit ONE frank-small request.
The 151,552 slot window could not hold it; the native window does.

No spec-decode interaction: checkpoint ships zero mtp.* tensors (config
lies), seam keeps every --speculative-* flag away from the engine.
This commit is contained in:
Frank
2026-10-01 14:03:25 -07:00
parent 389b605baf
commit 35e24db7e6
4 changed files with 35 additions and 21 deletions
+16 -10
View File
@@ -28,8 +28,10 @@
# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
# take most of it after boot, and eager-prefill activations then OOM on
# the first long request. The pool is therefore PINNED at 606,208 tokens
# (4 x 151,552), which leaves ~0.8 GB for the prefill workspace; a 170k
# single request and 4 concurrent 54k requests are verified against it.
# (2 x 262,144 native slots since the 2026-10-01 reslice; 4 x 151,552
# before), which leaves ~0.8 GB for the prefill workspace; a 170k single
# request and 4 concurrent 54k requests were verified at the 4-slot
# shape.
# The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
#
# Sampling: greedy by default (the fleet contract); a client that sends a
@@ -54,14 +56,17 @@ ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771
# Four slots on fp8 KV. The pool is pinned (see SIZING above): 606,208
# tokens = 4 x 151,552, so the slot window is 151,552. A single request
# may still grow to the whole pool (a 170k prompt is verified); the grid
# is the four-way concurrency contract.
# Two slots at the model's native 262,144 window on fp8 KV. The pool is
# pinned (see SIZING above): 606,208 tokens, and 2 x 262,144 = 524,288
# leaves ~82k tokens of pool slack. Resliced 2026-10-01 from 4 x 151,552
# so a compaction request (frank-primary's trigger 183,501 tokens + a
# 32,768-token summary budget = 216,269) fits ONE request; the grid is
# the two-way concurrency contract. A single request may still grow to
# the whole pool (a 170k prompt was verified at the 4-slot shape).
MEM_FRACTION=0.98
MAX_TOTAL_TOKENS=606208
CONTEXT_LENGTH=151552
MAX_RUNNING=4
CONTEXT_LENGTH=262144
MAX_RUNNING=2
KV_DTYPE=fp8_e4m3
QUANTIZATION=auto-round
# DeltaNet (mamba) recurrent states: the ssm state pool is the single
@@ -70,8 +75,9 @@ QUANTIZATION=auto-round
# slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
# bf16 dtype + 32 slots = 795 MiB.
MAMBA_SSM_DTYPE=bfloat16
# 32 slots is 8x the 4-way concurrency: enough for the radix mamba cache
# to keep shared-prefix states, small enough to leave the KV pool room.
# 32 slots is 16x the 2-way concurrency: enough for the radix mamba
# cache to keep shared-prefix states, small enough to leave the KV pool
# room.
MAX_MAMBA_CACHE_SIZE=32
# Reasoning is extracted into reasoning_content; tools use the qwen3_coder
+1 -1
View File
@@ -14,7 +14,7 @@
# WITHOUT speculative decoding — a NEXTN draft would be an uninitialised
# 4.24 GiB ghost model drafting at 0% acceptance.
#
# Context: CONTEXT_LENGTH x MAX_RUNNING=4 slots must fit the fp8 KV pool,
# Context: CONTEXT_LENGTH x MAX_RUNNING slots must fit the fp8 KV pool,
# which the engine autofits around the weights AND the DeltaNet ssm state
# pool (see frank-small.env). The mamba pool defaults to ~5.1 GB, so it is
# pinned (bf16, 32 slots). With the grid at its measured fit the card has