dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-small: three contexts, vision on, 2-3 GiB of the 3090 left free

Browse Source 
CTX_SIZE 589824 / PARALLEL 3 -> three independent 196,608-token conversations
(a mamba hybrid cannot share one KV cache, so per-slot window = pool/slots;
3 x 262,144 would need 18.4 GiB of KV alone and busts the card).

Vision comes back via bartowski's mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
(918,166,048 B, sha256 ff348f31..., fetched + verified by
scripts/download-frank-small.sh): the Q5_K_M GGUF ships no v.* tensors, so
--mmproj IS the image path. MMPROJ_FILE without a colon default so an explicit
empty value is a real text-only opt-out, and the deployer now gates on the
projector exactly as the serve script does.

Budget is measured, not guessed: the 2 x 262,144 shape used 20,190/24,576 MiB,
which prices non-KV cost at 7,390 MiB. Three slots + projector land near
22.5 GiB, i.e. ~2 GiB spare - the bottom of the requested band, to be
confirmed by measurement and laddered if it lands outside 2-3 GiB.
This commit is contained in:
Frank
2026-10-02 03:08:00 -07:00
parent 8f129180d8
commit a02850203d
6 changed files with 162 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/deploy-remote.sh
+12 -6
View File
@@ -1188,6 +1188,10 @@ deploy_llm_large() {
# not shared: frank-large's engine runs on the gfx1151 iGPU.
LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
# The projector IS the vision capability here: the Q5_K_M GGUF ships no `v.*`
# tensors, so a deploy that starts without this file silently ships a
# text-only endpoint. Checked unconditionally, like frank-large's.
LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
deploy_llm_small() {
# Refuse rather than restart-loop: a unit whose runtime or payload is
@@ -1200,12 +1204,14 @@ deploy_llm_small() {
echo "frank-small: && cmake --build build-cuda --target llama-server) first." >&2
return 1
fi
if [[ ! -f "$LLM_SMALL_MODEL_FILE" ]]; then
echo "ERROR: payload GGUF absent at $LLM_SMALL_MODEL_FILE" >&2
echo "frank-small: fetch bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M first;" >&2
echo "frank-small: refusing to start a unit that cannot load." >&2
return 1
fi
for f in "$LLM_SMALL_MODEL_FILE" "$LLM_SMALL_MMPROJ"; do
if [[ ! -f "$f" ]]; then
echo "ERROR: frank-small payload file absent at $f" >&2
echo "frank-small: run scripts/download-frank-small.sh first;" >&2
echo "frank-small: refusing to start a unit that cannot load." >&2
return 1
fi
done
relink_unit frank-small
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-small.service