frank-small: three contexts, vision on, 2-3 GiB of the 3090 left free

CTX_SIZE 589824 / PARALLEL 3 -> three independent 196,608-token conversations
(a mamba hybrid cannot share one KV cache, so per-slot window = pool/slots;
3 x 262,144 would need 18.4 GiB of KV alone and busts the card).

Vision comes back via bartowski's mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
(918,166,048 B, sha256 ff348f31..., fetched + verified by
scripts/download-frank-small.sh): the Q5_K_M GGUF ships no v.* tensors, so
--mmproj IS the image path. MMPROJ_FILE without a colon default so an explicit
empty value is a real text-only opt-out, and the deployer now gates on the
projector exactly as the serve script does.

Budget is measured, not guessed: the 2 x 262,144 shape used 20,190/24,576 MiB,
which prices non-KV cost at 7,390 MiB. Three slots + projector land near
22.5 GiB, i.e. ~2 GiB spare - the bottom of the requested band, to be
confirmed by measurement and laddered if it lands outside 2-3 GiB.
This commit is contained in:
Frank committed 2026-10-02 03:08:00 -07:00
1 parent 8f129180d8
commit a02850203d
6 files changed
+162 -81

No files matched your search

+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 2x262144, f16 K + q8_0 V) on the RTX 3090 (dankclaw, frank-small)
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 3x196608, f16 K + q8_0 V, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
After=network-online.target
Wants=network-online.target