frank-small: two wide unquantized contexts (2 x 200,704) on the 3090

Frank's order, two messages: "extend the max context of frank-small to
200700 per individual stream without further quantization", then
"reduce it to 2 streams for a total of 401400-ish tokens, set ubatch to
1k", then "unquantize the KV if we have extra space".

CTX_SIZE 540672 -> 401408 with PARALLEL 3 -> 2 gives n_ctx_slot 200,704:
above the 200,700 the harness compacts at, so a full-length request
lands in one stream instead of a 400 exceed_context_size_error. The pool
must divide evenly across slots and stay 256-aligned for the FA path,
which is why it is 401,408 and not 401,400.

CACHE_TYPE_V q8_0 -> f16: the freed third slot is exactly what pays for
unquantized values (24.5 -> 32 KiB/token), and the budget lands back in
the 2-3 GiB band Frank set on 2026-10-02 instead of leaving ~6 GiB of
the card idle. The asymmetric-KV contract that created this endpoint is
spent with it; both --cache-type flags stay explicit.

--batch-size/--ubatch-size are now passed (2048/1024). The endpoint had
never passed either, so it ran on engine defaults nobody wrote down.

Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md

scripts/frank-small.env

@@ -1,32 +1,42 @@
 # frank-small — llama.cpp (master, CUDA sm_86) serving
 # bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
 #
-# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to
-# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot
-# window shrinks — see the budget below.
+# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
+# tokens per conversation, above the 200,700 the harness compacts at — and the
+# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same
+# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given
+# up is exactly the VRAM that pays for unquantized V. No further quantization
+# anywhere on this endpoint.
 #
-# Two measurements on this card, this runtime, price the terms:
+# Three measurements on this card, this runtime, price the terms:
 #   2 x 262,144, no projector: 20,190 MiB used / 4,386 free
 #     KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
 #     DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
 #   3 x 196,608, projector on: 22,972 MiB used / 1,153 free
 #     KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
-# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
-# per slot left only 1.1 GiB free — outside the band. The pool therefore is
-# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted.
-# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and
-# the number the bats budget gate reproduces (it prices non-KV at 9 GiB).
+#   3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02
+#     shape) => non-KV = 8,748 MiB
+# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V
+# (32 KiB/token) this pool is 12,544 MiB, so the prediction is
+# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02
+# shape hit, bought with precision instead of a slot. The 1024 ubatch
+# re-prices the compute buffers upward, so MEASURE after any change: if the
+# boot lands under 2 GiB free, step the pool down in 12,288-token
+# (384 MiB at this rate) increments.
 #
 # KV arithmetic: attention KV exists only in the 8 full-attention layers
-# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token
-# (34 B per 32-element block) => 24.5 KiB/token, so 12,288 tokens (=4096/slot)
-# is one 293 MiB step — that is the ladder to walk if this moves again. The
-# native 262,144 per slot is NOT reachable three times: 3 x 262,144 x
-# 24.5 KiB = 18.4 GiB of KV before anything else is loaded.
+# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for
+# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that
+# is the ladder to walk if this moves again. Two native windows (2 x 262,144)
+# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV
+# against a card with ~15.7 GiB of KV room once weights, the vision tower, the
+# DeltaNet states and the graphs are loaded. 200,704 per stream is what full
+# precision buys; q8_0 on V would buy 262,144 instead.
 #
-# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token —
-# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is
-# the tensor the asymmetric-KV contract exists to protect.
+# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01
+# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit,
+# and the third slot is gone. The flags stay explicit so the choice is visible
+# in the argv either way.
 LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
 MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
 
@@ -41,20 +51,28 @@ ALIAS=frank-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771
 
-# Three independent conversations. llama.cpp divides --ctx-size across
+# Two independent conversations. llama.cpp divides --ctx-size across
 # --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
 # cannot use unified KV, so each slot gets its own cache and its own window.
-# 180,224 per slot is BELOW the native 262,144, and that is the cost of the
-# third slot: a longer single prompt needs compaction instead of one shot.
-# Consequence worth knowing: frank-primary's compaction trigger is 183,501
-# tokens, which no longer fits a frank-small slot — see the runbook's
-# "three slots vs compaction" note.
-CTX_SIZE=540672
-PARALLEL=3
+# 200,704 per slot is BELOW the native 262,144 — the price of paying for the
+# window in full precision — and ABOVE the harness's compaction trigger, which
+# is the point: a full-length request now lands in one stream instead of a 400
+# exceed_context_size_error. Cost of the third slot going away: a third
+# concurrent request queues behind a released slot instead of running.
+CTX_SIZE=401408
+PARALLEL=2
 
-# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
-# q8_0 on V is only served by the flash-attention path; the serve script
-# pins --flash-attn on and must never lose it.
+# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
+# so it ran on llama.cpp's own defaults (2048/512); the fleet now asks for a
+# 1024 ubatch on this shape. Both are tunables because the pair is a
+# measurement, not a law: on the 27B sibling 512 was the prefill peak and every
+# step up cost GiB of compute buffer for no speed.
+BATCH_SIZE=2048
+UBATCH_SIZE=1024
+
+# The KV contract (Frank, 2026-10-03): unquantized K AND V. --flash-attn stays
+# pinned on in the serve script — it is what made the q8_0-V path cheap and it
+# remains the fast path for f16 KV.
 CACHE_TYPE_K=f16
-CACHE_TYPE_V=q8_0
+CACHE_TYPE_V=f16
 NGL=99

scripts/frank-small.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 3x180224, f16 K + q8_0 V, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
+Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 2x200704, unquantized f16 KV, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
 After=network-online.target
 Wants=network-online.target
 

scripts/serve-frank-small.sh

@@ -4,10 +4,10 @@
 #
 # 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
 # safetensors to the llama.cpp GGUF path, because llama.cpp is the only
-# runtime on this host that can serve the MiMo qwen35 hybrid with
-# ASYMMETRIC KV: --cache-type-k f16 (unquantized keys) + --cache-type-v
-# q8_0 (quantized values), which is Frank's explicit KV contract for this
-# endpoint. SGLang's --kv-cache-dtype only moves K and V together.
+# runtime on this host that moves K and V independently — the asymmetric
+# f16-K + q8_0-V contract that endpoint was built for. 2026-10-03 (Frank):
+# the pair is now f16/f16 — two wide slots freed the VRAM, so the values are
+# unquantized too. The per-tensor flags stay explicit either way.
 #
 # Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
 # DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
@@ -37,14 +37,20 @@ fi
 : "${ALIAS:=frank-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7771}"
-# ctx-size is the POOL llama.cpp divides across slots: 540672 / 3 gives
-# 180224 per slot — three independent conversations (the native 262144 no
-# longer fits three times on a 24 GiB card once the vision tower is loaded;
-# frank-small.env carries the two measurements that set this number).
-: "${CTX_SIZE:=540672}"
-: "${PARALLEL:=3}"
+# ctx-size is the POOL llama.cpp divides across slots: 401408 / 2 gives
+# 200,704 per slot — two independent conversations, each one wide enough to
+# hold a whole frank-primary-sized prompt in a single shot (the native 262,144
+# no longer fits twice on a 24 GiB card once the vision tower is loaded;
+# frank-small.env carries the three measurements that set this number).
+: "${CTX_SIZE:=401408}"
+: "${PARALLEL:=2}"
+# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either
+# flag, so it ran on llama.cpp's own defaults (2048/512); they are tunables now
+# because the 1024 ubatch is what this shape was booted with.
+: "${BATCH_SIZE:=2048}"
+: "${UBATCH_SIZE:=1024}"
 : "${CACHE_TYPE_K:=f16}"
-: "${CACHE_TYPE_V:=q8_0}"
+: "${CACHE_TYPE_V:=f16}"
 : "${NGL:=99}"
 # Vision projector. No colon on purpose: an explicitly empty value means
 # "text-only, on purpose" and must NOT be replaced by the default path.
@@ -66,9 +72,10 @@ fi
 # open after boot; make sure it exists before the server needs it.
 nvidia-modprobe -u -c=0 2>/dev/null || true
 
-# q8_0 on V REQUIRES flash attention (the quantized-value kernels only exist
-# on the FA path); an FA-less build silently falls back and the KV contract
-# in frank-small.env becomes a lie, so pin the flag pair here.
+# Flash attention stays pinned on: it is the fast path for f16 KV, and it is
+# the only path that could serve quantized V at all — so if CACHE_TYPE_V is
+# ever set back to q8_0 this flag must not be lost, or the build falls back
+# silently and the KV contract in frank-small.env becomes a lie.
 ARGS=(
     --model "$MODEL_FILE"
     --alias "$ALIAS"
@@ -76,6 +83,8 @@ ARGS=(
     --port "$LISTEN_PORT"
     --ctx-size "$CTX_SIZE"
     --parallel "$PARALLEL"
+    --batch-size "$BATCH_SIZE"
+    --ubatch-size "$UBATCH_SIZE"
     --n-gpu-layers "$NGL"
     --flash-attn on
     --cache-type-k "$CACHE_TYPE_K"