frank-small: Q5_K_M GGUF on llama.cpp, two full native windows

One quant step down from Q6_K on the same CUDA build and the same shape:
2 x 262,144 with K f16 + V q8_0. The 0.86 GiB freed from the weights is
headroom, not a new window — the native 262,144 cap is already reached,
and f16 V would still cost 3.96 GiB more than the pool can carry.
This commit is contained in:
Frank committed 2026-10-02 00:00:17 -07:00
1 parent 4bb10cd20e
commit d82c5c03b7
5 files changed
+36 -26

No files matched your search

+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q6_K) as frank-small on
# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q5_K_M) as frank-small on
# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
#
# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
@@ -32,7 +32,7 @@ fi
# --- defaults (used when the env file does not set them) -------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
: "${MODEL_FILE:=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf}"
: "${MODEL_FILE:=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
: "${ALIAS:=frank-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7771}"