frank-small: reslice to 2 x 262,144 native slots (was 4 x 151,552)
The KV pool stays pinned at 606,208 tokens (card is full: 531 MiB free at the old shape; autofit's 687,388 would leave 0.53 GB and OOM on the first long request). 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack. Motivation: harness compaction. frank-primary's contextWindow 262,144 compacts at 70% = 183,501 tokens, and the compaction request carries a 32,768-token summary budget -> 216,269 must fit ONE frank-small request. The 151,552 slot window could not hold it; the native window does. No spec-decode interaction: checkpoint ships zero mtp.* tensors (config lies), seam keeps every --speculative-* flag away from the engine.
This commit is contained in:
+16
-10
@@ -28,8 +28,10 @@
|
||||
# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
|
||||
# take most of it after boot, and eager-prefill activations then OOM on
|
||||
# the first long request. The pool is therefore PINNED at 606,208 tokens
|
||||
# (4 x 151,552), which leaves ~0.8 GB for the prefill workspace; a 170k
|
||||
# single request and 4 concurrent 54k requests are verified against it.
|
||||
# (2 x 262,144 native slots since the 2026-10-01 reslice; 4 x 151,552
|
||||
# before), which leaves ~0.8 GB for the prefill workspace; a 170k single
|
||||
# request and 4 concurrent 54k requests were verified at the 4-slot
|
||||
# shape.
|
||||
# The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract); a client that sends a
|
||||
@@ -54,14 +56,17 @@ ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
|
||||
# Four slots on fp8 KV. The pool is pinned (see SIZING above): 606,208
|
||||
# tokens = 4 x 151,552, so the slot window is 151,552. A single request
|
||||
# may still grow to the whole pool (a 170k prompt is verified); the grid
|
||||
# is the four-way concurrency contract.
|
||||
# Two slots at the model's native 262,144 window on fp8 KV. The pool is
|
||||
# pinned (see SIZING above): 606,208 tokens, and 2 x 262,144 = 524,288
|
||||
# leaves ~82k tokens of pool slack. Resliced 2026-10-01 from 4 x 151,552
|
||||
# so a compaction request (frank-primary's trigger 183,501 tokens + a
|
||||
# 32,768-token summary budget = 216,269) fits ONE request; the grid is
|
||||
# the two-way concurrency contract. A single request may still grow to
|
||||
# the whole pool (a 170k prompt was verified at the 4-slot shape).
|
||||
MEM_FRACTION=0.98
|
||||
MAX_TOTAL_TOKENS=606208
|
||||
CONTEXT_LENGTH=151552
|
||||
MAX_RUNNING=4
|
||||
CONTEXT_LENGTH=262144
|
||||
MAX_RUNNING=2
|
||||
KV_DTYPE=fp8_e4m3
|
||||
QUANTIZATION=auto-round
|
||||
# DeltaNet (mamba) recurrent states: the ssm state pool is the single
|
||||
@@ -70,8 +75,9 @@ QUANTIZATION=auto-round
|
||||
# slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
|
||||
# bf16 dtype + 32 slots = 795 MiB.
|
||||
MAMBA_SSM_DTYPE=bfloat16
|
||||
# 32 slots is 8x the 4-way concurrency: enough for the radix mamba cache
|
||||
# to keep shared-prefix states, small enough to leave the KV pool room.
|
||||
# 32 slots is 16x the 2-way concurrency: enough for the radix mamba
|
||||
# cache to keep shared-prefix states, small enough to leave the KV pool
|
||||
# room.
|
||||
MAX_MAMBA_CACHE_SIZE=32
|
||||
|
||||
# Reasoning is extracted into reasoning_content; tools use the qwen3_coder
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
# WITHOUT speculative decoding — a NEXTN draft would be an uninitialised
|
||||
# 4.24 GiB ghost model drafting at 0% acceptance.
|
||||
#
|
||||
# Context: CONTEXT_LENGTH x MAX_RUNNING=4 slots must fit the fp8 KV pool,
|
||||
# Context: CONTEXT_LENGTH x MAX_RUNNING slots must fit the fp8 KV pool,
|
||||
# which the engine autofits around the weights AND the DeltaNet ssm state
|
||||
# pool (see frank-small.env). The mamba pool defaults to ~5.1 GB, so it is
|
||||
# pinned (bf16, 32 slots). With the grid at its measured fit the card has
|
||||
|
||||
Reference in New Issue
Block a user