frank-small: two wide unquantized contexts (2 x 200,704) on the 3090

Frank's order, two messages: "extend the max context of frank-small to
200700 per individual stream without further quantization", then
"reduce it to 2 streams for a total of 401400-ish tokens, set ubatch to
1k", then "unquantize the KV if we have extra space".

CTX_SIZE 540672 -> 401408 with PARALLEL 3 -> 2 gives n_ctx_slot 200,704:
above the 200,700 the harness compacts at, so a full-length request
lands in one stream instead of a 400 exceed_context_size_error. The pool
must divide evenly across slots and stay 256-aligned for the FA path,
which is why it is 401,408 and not 401,400.

CACHE_TYPE_V q8_0 -> f16: the freed third slot is exactly what pays for
unquantized values (24.5 -> 32 KiB/token), and the budget lands back in
the 2-3 GiB band Frank set on 2026-10-02 instead of leaving ~6 GiB of
the card idle. The asymmetric-KV contract that created this endpoint is
spent with it; both --cache-type flags stay explicit.

--batch-size/--ubatch-size are now passed (2048/1024). The endpoint had
never passed either, so it ran on engine defaults nobody wrote down.

Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md

scripts/serve-frank-small.sh

@@ -4,10 +4,10 @@
 #
 # 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
 # safetensors to the llama.cpp GGUF path, because llama.cpp is the only
-# runtime on this host that can serve the MiMo qwen35 hybrid with
-# ASYMMETRIC KV: --cache-type-k f16 (unquantized keys) + --cache-type-v
-# q8_0 (quantized values), which is Frank's explicit KV contract for this
-# endpoint. SGLang's --kv-cache-dtype only moves K and V together.
+# runtime on this host that moves K and V independently — the asymmetric
+# f16-K + q8_0-V contract that endpoint was built for. 2026-10-03 (Frank):
+# the pair is now f16/f16 — two wide slots freed the VRAM, so the values are
+# unquantized too. The per-tensor flags stay explicit either way.
 #
 # Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
 # DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
@@ -37,14 +37,20 @@ fi
 : "${ALIAS:=frank-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7771}"
-# ctx-size is the POOL llama.cpp divides across slots: 540672 / 3 gives
-# 180224 per slot — three independent conversations (the native 262144 no
-# longer fits three times on a 24 GiB card once the vision tower is loaded;
-# frank-small.env carries the two measurements that set this number).
-: "${CTX_SIZE:=540672}"
-: "${PARALLEL:=3}"
+# ctx-size is the POOL llama.cpp divides across slots: 401408 / 2 gives
+# 200,704 per slot — two independent conversations, each one wide enough to
+# hold a whole frank-primary-sized prompt in a single shot (the native 262,144
+# no longer fits twice on a 24 GiB card once the vision tower is loaded;
+# frank-small.env carries the three measurements that set this number).
+: "${CTX_SIZE:=401408}"
+: "${PARALLEL:=2}"
+# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either
+# flag, so it ran on llama.cpp's own defaults (2048/512); they are tunables now
+# because the 1024 ubatch is what this shape was booted with.
+: "${BATCH_SIZE:=2048}"
+: "${UBATCH_SIZE:=1024}"
 : "${CACHE_TYPE_K:=f16}"
-: "${CACHE_TYPE_V:=q8_0}"
+: "${CACHE_TYPE_V:=f16}"
 : "${NGL:=99}"
 # Vision projector. No colon on purpose: an explicitly empty value means
 # "text-only, on purpose" and must NOT be replaced by the default path.
@@ -66,9 +72,10 @@ fi
 # open after boot; make sure it exists before the server needs it.
 nvidia-modprobe -u -c=0 2>/dev/null || true
 
-# q8_0 on V REQUIRES flash attention (the quantized-value kernels only exist
-# on the FA path); an FA-less build silently falls back and the KV contract
-# in frank-small.env becomes a lie, so pin the flag pair here.
+# Flash attention stays pinned on: it is the fast path for f16 KV, and it is
+# the only path that could serve quantized V at all — so if CACHE_TYPE_V is
+# ever set back to q8_0 this flag must not be lost, or the build falls back
+# silently and the KV contract in frank-small.env becomes a lie.
 ARGS=(
     --model "$MODEL_FILE"
     --alias "$ALIAS"
@@ -76,6 +83,8 @@ ARGS=(
     --port "$LISTEN_PORT"
     --ctx-size "$CTX_SIZE"
     --parallel "$PARALLEL"
+    --batch-size "$BATCH_SIZE"
+    --ubatch-size "$UBATCH_SIZE"
     --n-gpu-layers "$NGL"
     --flash-attn on
     --cache-type-k "$CACHE_TYPE_K"