frank-small: three contexts, vision on, 2-3 GiB of the 3090 left free

CTX_SIZE 589824 / PARALLEL 3 -> three independent 196,608-token conversations
(a mamba hybrid cannot share one KV cache, so per-slot window = pool/slots;
3 x 262,144 would need 18.4 GiB of KV alone and busts the card).

Vision comes back via bartowski's mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
(918,166,048 B, sha256 ff348f31..., fetched + verified by
scripts/download-frank-small.sh): the Q5_K_M GGUF ships no v.* tensors, so
--mmproj IS the image path. MMPROJ_FILE without a colon default so an explicit
empty value is a real text-only opt-out, and the deployer now gates on the
projector exactly as the serve script does.

Budget is measured, not guessed: the 2 x 262,144 shape used 20,190/24,576 MiB,
which prices non-KV cost at 7,390 MiB. Three slots + projector land near
22.5 GiB, i.e. ~2 GiB spare - the bottom of the requested band, to be
confirmed by measurement and laddered if it lands outside 2-3 GiB.

scripts/serve-frank-small.sh

@@ -11,9 +11,10 @@
 #
 # Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
 # DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
-# the DeltaNet state is per-slot fixed size. GGUF path is TEXT-ONLY: no
-# mmproj, the in-checkpoint vision tower does not exist here (models.yml
-# declares frank-small text-only since this swap).
+# the DeltaNet state is per-slot fixed size. Vision comes from the repo's
+# separate projector (the Q5_K_M GGUF carries no `v.*` tensors at all), so
+# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
+# text-only opt-out.
 #
 # Sampling: llama.cpp takes it per request (the fleet harness sends its
 # recipe from models.yml); greedy is the harness default.
@@ -36,20 +37,30 @@ fi
 : "${ALIAS:=frank-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7771}"
-# ctx-size is the POOL llama.cpp divides across slots: 524288 / 2 = TWO
-# full native 262144 windows.
-: "${CTX_SIZE:=524288}"
-: "${PARALLEL:=2}"
+# ctx-size is the POOL llama.cpp divides across slots: 589824 / 3 gives
+# 196608 per slot — three independent conversations, each with a 192k window
+# (the native 262144 no longer fits three times on a 24 GiB card; see
+# frank-small.env for the arithmetic that lands ~2.5 GiB of card free).
+: "${CTX_SIZE:=589824}"
+: "${PARALLEL:=3}"
 : "${CACHE_TYPE_K:=f16}"
 : "${CACHE_TYPE_V:=q8_0}"
 : "${NGL:=99}"
+# Vision projector. No colon on purpose: an explicitly empty value means
+# "text-only, on purpose" and must NOT be replaced by the default path.
+: "${MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
 
 # Fail loudly rather than half-loading: a missing file is a download that
-# has not finished, and Restart=on-failure would turn it into a crash loop
-# that reads like a runtime fault.
+# has not finished, and Restart=always would turn it into a crash loop that
+# reads like a runtime fault.
 for path in "$LLAMA_BIN" "$MODEL_FILE"; do
     [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
 done
+EXTRA_ARGS=()
+if [[ -n "$MMPROJ_FILE" ]]; then
+    [[ -e "$MMPROJ_FILE" ]] || { echo "frank-small: missing $MMPROJ_FILE" >&2; exit 1; }
+    EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
+fi
 
 # The UVM device node is created lazily by nvidia-modprobe on the first CUDA
 # open after boot; make sure it exists before the server needs it.
@@ -58,17 +69,19 @@ nvidia-modprobe -u -c=0 2>/dev/null || true
 # q8_0 on V REQUIRES flash attention (the quantized-value kernels only exist
 # on the FA path); an FA-less build silently falls back and the KV contract
 # in frank-small.env becomes a lie, so pin the flag pair here.
-exec "$LLAMA_BIN" \
-    --model "$MODEL_FILE" \
-    --alias "$ALIAS" \
-    --host "$BIND_ADDR" \
-    --port "$LISTEN_PORT" \
-    --ctx-size "$CTX_SIZE" \
-    --parallel "$PARALLEL" \
-    --n-gpu-layers "$NGL" \
-    --flash-attn on \
-    --cache-type-k "$CACHE_TYPE_K" \
-    --cache-type-v "$CACHE_TYPE_V" \
-    --jinja \
-    --reasoning auto \
+ARGS=(
+    --model "$MODEL_FILE"
+    --alias "$ALIAS"
+    --host "$BIND_ADDR"
+    --port "$LISTEN_PORT"
+    --ctx-size "$CTX_SIZE"
+    --parallel "$PARALLEL"
+    --n-gpu-layers "$NGL"
+    --flash-attn on
+    --cache-type-k "$CACHE_TYPE_K"
+    --cache-type-v "$CACHE_TYPE_V"
+    --jinja
+    --reasoning auto
     --reasoning-format deepseek
+)
+exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}