dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: Q8_0 model, 160k single slot, q6_0 KV + Hadamard

Browse Source 
Switch to CRACK Q8_0 model (37.8 GB, mmproj f16). Servt as a single
160000-ctx slot (PARALLEL=1); speculative decoding stays disabled. KV
cache is q6_0 + Hadamard. iq5_k is not a valid KV-cache type in ik_llama
(cache whitelist excludes iq5_*); q8_KV reproduces the ggml.c:5427 view
assert even at PARALLEL=1/no-spec, so q6_0 is the verified working 6-bit
choice. Env, serve script, tests, and KB updated. Deployment was left
down after the isolation test; this commit redeploys it with the
verified Q8_0/160k/q6_0 config.
This commit is contained in:
Frank committed 2026-08-26 05:01:58 -07:00
1 parent 23e905c712
commit 8c41c821c4
4 files changed
+67 -75
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/frank_small_config.bats
+15 -11
View File
@@ -9,15 +9,21 @@ SERVE_FILE="$REPO_ROOT/scripts/serve-frank-small.sh"
UNIT_FILE="$REPO_ROOT/scripts/frank-default-small.service"
TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
@test "frank-default-small selects the Ornith Q6_K model" {
grep -q '^MODEL_FILE=Ornith-1\.5-35B-A3B-CRACK-Q6_K\.gguf$' "$ENV_FILE"
grep -q 'Ornith-1\.5-35B-A3B-CRACK-Q6_K\.gguf' "$SERVE_FILE"
@test "frank-default-small selects the Ornith Q8_0 model" {
grep -q '^MODEL_FILE=Ornith-1\.5-35B-A3B-CRACK-Q8_0\.gguf$' "$ENV_FILE"
grep -q 'Ornith-1\.5-35B-A3B-CRACK-Q8_0\.gguf' "$SERVE_FILE"
}
@test "frank-default-small uses 163840 context per slot across three slots" {
grep -q '^CTX_SIZE=491520$' "$ENV_FILE"
grep -q '163840 ctx each' "$ENV_FILE"
grep -q '^PARALLEL=3$' "$ENV_FILE"
@test "frank-default-small uses a single 160000 context slot" {
grep -q '^CTX_SIZE=160000$' "$ENV_FILE"
grep -q '^PARALLEL=1$' "$ENV_FILE"
}
@test "frank-default-small uses q6_0 KV cache with Hadamard" {
grep -q '^CACHE_TYPE_K=q6_0$' "$ENV_FILE"
grep -q '^CACHE_TYPE_V=q6_0$' "$ENV_FILE"
grep -q '^K_CACHE_HADAMARD=1$' "$ENV_FILE"
grep -q '^V_CACHE_HADAMARD=1$' "$ENV_FILE"
}
@test "frank-default-small keeps full-attention and final blocks on CUDA" {
@@ -40,11 +46,9 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
[[ "$N_GPU_LAYERS" == 999 ]]
}
@test "frank-default-small disables speculative decoding for 3 parallel slots" {
# ik_llama limits speculative decoding to -np 1 and hard-exits with
# --parallel > 1; this service keeps 3 parallel slots and runs SPEC_TYPE=none.
@test "frank-default-small disables speculative decoding (single slot)" {
grep -q '^SPEC_TYPE=none$' "$ENV_FILE"
grep -q '^PARALLEL=3$' "$ENV_FILE"
grep -q '^PARALLEL=1$' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"