dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-small: Q5_K_M GGUF on llama.cpp, two full native windows

Browse Source 
One quant step down from Q6_K on the same CUDA build and the same shape:
2 x 262,144 with K f16 + V q8_0. The 0.86 GiB freed from the weights is
headroom, not a new window — the native 262,144 cap is already reached,
and f16 V would still cost 3.96 GiB more than the pool can carry.
This commit is contained in:
Frank committed 2026-10-02 00:00:17 -07:00
1 parent 4bb10cd20e
commit d82c5c03b7
5 files changed
+36 -26
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-small.service
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (qwen35 hybrid, 2x262144, f16 K + q8_0 V) on the RTX 3090 (dankclaw, frank-small)
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 2x262144, f16 K + q8_0 V) on the RTX 3090 (dankclaw, frank-small)
After=network-online.target
Wants=network-online.target