frank-small: Q5_K_M GGUF on llama.cpp, two full native windows
One quant step down from Q6_K on the same CUDA build and the same shape: 2 x 262,144 with K f16 + V q8_0. The 0.86 GiB freed from the weights is headroom, not a new window — the native 262,144 cap is already reached, and f16 V would still cost 3.96 GiB more than the pool can carry.
This commit is contained in:
1 parent
4bb10cd20e
commit
d82c5c03b7
5 files changed
+36
-26
No files matched your search
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q6_K) as frank-small on
|
||||
# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q5_K_M) as frank-small on
|
||||
# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
||||
#
|
||||
# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
|
||||
@@ -32,7 +32,7 @@ fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_FILE:=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf}"
|
||||
: "${MODEL_FILE:=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf}"
|
||||
: "${ALIAS:=frank-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
|
||||
Reference in new issue
Block a user