frank-small: three contexts, vision on, 2-3 GiB of the 3090 left free
CTX_SIZE 589824 / PARALLEL 3 -> three independent 196,608-token conversations (a mamba hybrid cannot share one KV cache, so per-slot window = pool/slots; 3 x 262,144 would need 18.4 GiB of KV alone and busts the card). Vision comes back via bartowski's mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf (918,166,048 B, sha256 ff348f31..., fetched + verified by scripts/download-frank-small.sh): the Q5_K_M GGUF ships no v.* tensors, so --mmproj IS the image path. MMPROJ_FILE without a colon default so an explicit empty value is a real text-only opt-out, and the deployer now gates on the projector exactly as the serve script does. Budget is measured, not guessed: the 2 x 262,144 shape used 20,190/24,576 MiB, which prices non-KV cost at 7,390 MiB. Three slots + projector land near 22.5 GiB, i.e. ~2 GiB spare - the bottom of the requested band, to be confirmed by measurement and laddered if it lands outside 2-3 GiB.
This commit is contained in:
1 parent
8f129180d8
commit
a02850203d
6 files changed
+162
-81
No files matched your search
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 2x262144, f16 K + q8_0 V) on the RTX 3090 (dankclaw, frank-small)
|
||||
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 3x196608, f16 K + q8_0 V, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
Reference in new issue
Block a user