frank-default-small: use split-mode none on single-GPU dankserver

--split-mode graph requires >= 2 devices in ik_llama.cpp and errors hard
at model load on a single RTX 3090 ("It is not possible to use split mode
'graph' with less than 2 devices"), looping the service. Set SPLIT_MODE
=none (the default), which works with the --override-tensor CPU expert
offload. KB updated with the two-device requirement so we do not assume
graph degrades gracefully on one GPU.
This commit is contained in:
Frank
2026-08-26 03:06:30 -07:00
parent b8de25d70b
commit e2085f1f7d
5 changed files with 25 additions and 23 deletions
+5 -3
View File
@@ -76,9 +76,11 @@ CACHE_TYPE_V=q8_KV
K_CACHE_HADAMARD=1
V_CACHE_HADAMARD=1
# Split mode: graph (multi-GPU graph split; on a single 3090 this is a no-op
# for correctness but kept per operator preference).
SPLIT_MODE=graph
# Split mode: none. Dankserver has a single RTX 3090 and ik_llama requires
# >= 2 devices for split-mode graph ('It is not possible to use split mode
# graph with less than 2 devices' is a hard load error). none is correct
# here and still works with OVERRIDE_EXPERT CPU expert offload.
SPLIT_MODE=none
VERBOSITY=4
# Store context checkpoints in RAM to hold peak GPU memory down.
+2 -4
View File
@@ -10,10 +10,8 @@
# KV cache: q8_KV with Hadamard transforms on K and V (requires a build with
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
#
# Split mode graph + --override-tensor CPU expert offload. Per ik_llama
# README, `--split-mode graph` + partial GPU offload can occasionally produce
# incoherent output; the documented fallback is `-cuda graphs=0`. Smoke-test
# after each model load.
# Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
# split-mode graph with fewer than 2 devices (hard load error).
#
# Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
# --override-tensor pattern are pinned to CPU. Selected full-attention and