dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: run on ik_llama.cpp with q8_KV + Hadamard cache

Browse Source 
Switch frank-default-small on Dankserver from upstream llama.cpp (b10121)
to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13
toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the
old binary is kept for rollback.

Runtime changes:
- KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard
- --split-mode graph (single-GPU no-op, retained per operator pref)
- migrate ngram speculation to canonical
  --spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24
  (legacy --spec-ngram-mod-* are rejected in ik main)
- N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay
  pinned to CPU via --override-tensor

Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory
for the quantized KV cache; recipe + reasoning baked into
docs/LLM/ikllama-ornith.md. Tests and both indices updated.
This commit is contained in:
Frank
2026-08-26 02:59:34 -07:00
parent c8d7ed65c6
commit b8de25d70b
6 changed files with 295 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_small_config.bats
+11 -6
View File
@@ -35,17 +35,22 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
[[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
[[ "$OVERRIDE_BUFT" == CPU ]]
[[ "$N_GPU_LAYERS" == all ]]
# ik_llama --gpu-layers/-ngl requires an integer ("all" is rejected); 999
# loads as many layers as possible while OVERRIDE_EXPERT pins exps to CPU.
[[ "$N_GPU_LAYERS" == 999 ]]
}
@test "frank-default-small uses ngram-mod without MTP" {
grep -q '^SPEC_TYPE=ngram-mod$' "$ENV_FILE"
grep -q '^SPEC_NGRAM_MOD_N_MIN=48$' "$ENV_FILE"
grep -q '^SPEC_NGRAM_MOD_N_MAX=64$' "$ENV_FILE"
grep -q '^SPEC_NGRAM_MOD_N_MATCH=24$' "$ENV_FILE"
# ik_llama main uses the canonical --spec-type ngram-mod:<k=v> form; the
# legacy --spec-ngram-mod-n-* flags are rejected.
grep -q '^SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24$' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"
! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE"
! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE"
grep -q -- '--spec-ngram-mod-n-max' "$SERVE_FILE"
grep -q -- '--spec-type' "$SERVE_FILE"
! grep -q -- '--spec-ngram-mod-' "$SERVE_FILE"
}
@test "frank-default-small uses ubatch 1024" {