frank-default-small: render Qwen-Sharp via --jinja; pull more experts to GPU

--jinja: ik_llama only renders a custom --chat-template-file as real Jinja
when --jinja is passed (inputs.use_jinja ? apply_jinja : non-jinja
fallback). Without it the Qwen-Sharp template's thinking/image/tool
formatting is silently dropped. Add USE_JINJA=1 -> --jinja.

Experts: the RTX 3090 had ~5.4 GiB headroom; pull expert blocks 30,32,33,34
back to CUDA (each block ~630 MiB) from CPU for lower latency. Update the
OVERRIDE_EXPERT regex, its test, and the KB with the headroom-check note.
This commit is contained in:
Frank
2026-08-26 03:46:54 -07:00
parent 4144f0f0e8
commit 3f3b1bdb27
4 changed files with 49 additions and 7 deletions
+26 -5
View File
@@ -1,7 +1,7 @@
---
type: reference
title: ik_llama.cpp on Dankserver (Ornith Q6_K)
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), speculative decoding disabled (3 parallel slots), and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), speculative decoding disabled (3 parallel slots), the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
tags:
- ikllama
- llamacpp
@@ -144,17 +144,38 @@ cache RAM vs f16.
> single-GPU host you must use `none` (or `layer`, which also needs >1).
> Do not assume `--split-mode graph` degrades gracefully on one GPU.
### Expert offload (MoE on CPU)
### Expert offload (MoE: CPU with a partial GPU slice)
The expert-offload pattern is unchanged from upstream llama.cpp: the
The expert-offload pattern from upstream llama.cpp carries over: the
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
On 2026-08-26 we brought expert blocks **30, 32, 33, 34 back to CUDA**
(~2.5 GiB) to use the ~5.4 GiB GPU headroom for lower latency; the rest of
that CPU range stays on CPU.
- `OVERRIDE_EXPERT` — the layer-bounded regex (same as before the switch).
- `OVERRIDE_EXPERT` — the layer-bounded regex listing only the blocks whose
experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
24|25|26|28|29)`. Blocks 30,32,33,34 (and the always-on full-attention +
35-40 set) are left off the list so their experts load to CUDA.
- `OVERRIDE_BUFT=CPU`.
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
integer (the literal `all` is rejected). 999 means "load as many tensor
layers as possible", while the expert override still pins the exps to CPU.
layers as possible", while the expert override still pins the listed exps
to CPU.
To move more experts back to the GPU later, drop more block numbers from
`OVERRIDE_EXPERT` (each block is ~630 MiB; verify `nvidia-smi` headroom
first and keep ~2-3 GiB margin for KV/compute).
### Qwen-Sharp chat template via --jinja
`--chat-template-file` alone only stores the template text; ik_llama **does
not** render a custom template as real Jinja unless `--jinja` is also given
(`inputs.use_jinja ? common_chat_templates_apply_jinja : <non-jinja
fallback>`). Without `--jinja` it warns it only supports "commonly used
templates" and silently uses built-in handlers, dropping the Qwen-Sharp
template's thinking / image / tool formatting. So the serve script passes
both `--chat-template-file` and `--jinja` (gated on `USE_JINJA`, set to `1`).
### Speculative decoding: disabled (3 parallel slots)