--split-mode graph requires >= 2 devices in ik_llama.cpp and errors hard
at model load on a single RTX 3090 ("It is not possible to use split mode
'graph' with less than 2 devices"), looping the service. Set SPLIT_MODE
=none (the default), which works with the --override-tensor CPU expert
offload. KB updated with the two-device requirement so we do not assume
graph degrades gracefully on one GPU.
Switch frank-default-small on Dankserver from upstream llama.cpp (b10121)
to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13
toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the
old binary is kept for rollback.
Runtime changes:
- KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard
- --split-mode graph (single-GPU no-op, retained per operator pref)
- migrate ngram speculation to canonical
--spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24
(legacy --spec-ngram-mod-* are rejected in ik main)
- N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay
pinned to CPU via --override-tensor
Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory
for the quantized KV cache; recipe + reasoning baked into
docs/LLM/ikllama-ornith.md. Tests and both indices updated.
3 slots x 229376 (262144 - 32768) = 688128 total context. Updates the
.env, models.json (contextWindow/maxTokens 229376 per slot), and the
index/deployment/ornith runbook references.
- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled
native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an
explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU
(replaces --n-cpu-moe), Qwen-Sharp chat template.
- frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0
ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head
(--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the
retired PromptForge .pfs sidecar lanes.
- Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via
--chat-template-file on both services.
- Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages;
update index, deployment, and both system specs; models.json names.
Smoke-tested both models on the real hardware on scratch ports before
commit: model loads, MTP draft attaches, Sharp template applies, chat
completion returns.
The LLM fleet serve scripts, systemd units, env tunables, and bench
harnesses join the deploy scripts. The vllmScripts deploy.sh is
removed here; its functionality is folded into the unified
scripts/deploy.sh in a follow-up commit.
bench-spec-stack.sh gains the executable bit (required by the
scripts_syntax test seam).
Frank