dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-small: Q5_K_M GGUF on llama.cpp, two full native windows

Browse Source 
One quant step down from Q6_K on the same CUDA build and the same shape:
2 x 262,144 with K f16 + V q8_0. The 0.86 GiB freed from the weights is
headroom, not a new window — the native 262,144 cap is already reached,
and f16 V would still cost 3.96 GiB more than the pool can carry.
This commit is contained in:
Frank committed 2026-10-02 00:00:17 -07:00
1 parent 4bb10cd20e
commit d82c5c03b7
5 files changed
+36 -26
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/frank_small_unit.bats
+11 -7
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bats
# Seam: the frank-small llama.cpp root unit on dankclaw.
# frank-small is the RTX 3090 endpoint; since 2026-10-01 night it serves
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q6_K via a CUDA (sm_86) build of
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M via a CUDA (sm_86) build of
# llama.cpp master, chosen for the ASYMMETRIC KV Frank asked for: K f16
# (unquantized) + V q8_0, which only llama.cpp exposes per-tensor. This
# seam runs the serve script against a stub engine and asserts on the argv
@@ -14,7 +14,7 @@ SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
TARGET_STUB="MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf"
TARGET_STUB="MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf"
setup() {
STAGE="$(mktemp -d)"
@@ -72,9 +72,9 @@ argv_value() {
grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
}
@test "the shipped env points at the llama.cpp runtime and the Q6_K GGUF" {
@test "the shipped env points at the llama.cpp runtime and the Q5_K_M GGUF" {
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE"
grep -qE "^MODEL_FILE=/home/dank/models/MiMo-Q6K/$TARGET_STUB$" "$ENV_FILE"
grep -qE "^MODEL_FILE=/home/dank/models/MiMo-Q5KM/$TARGET_STUB$" "$ENV_FILE"
}
@test "the shipped env serves the fleet shape: two full native windows" {
@@ -105,12 +105,16 @@ argv_value() {
@test "the KV budget fits the 3090 with room for checkpoints" {
# K f16 (16 KiB/token) + V q8_0 (~8.5, budget 25 with slack) over the
# whole pool, plus 7.8 GiB weights and ~2 GiB of compute buffers and
# whole pool, plus 6.4 GiB weights and ~2 GiB of compute buffers and
# CUDA graphs, must stay under the card's 24 GiB. This is arithmetic,
# not a boot log: change CTX_SIZE and it bites before deploy does.
# not a boot log: change CTX_SIZE and it bites before deploy does. The
# same arithmetic with f16 V (16 KiB/token, so 32+2 overhead per token)
# lands past the card, which is why the contract is asymmetric.
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
kv_gib=$((ctx * 25 / 1048576))
[ $((kv_gib + 8 + 2)) -le 24 ]
[ $((kv_gib + 7 + 2)) -le 24 ]
f16v_gib=$((ctx * 34 / 1048576))
[ $((f16v_gib + 7 + 2)) -gt 24 ]
}
@test "no speculative draft ships by default" {