frank-default-small: render Qwen-Sharp via --jinja; pull more experts to GPU
--jinja: ik_llama only renders a custom --chat-template-file as real Jinja when --jinja is passed (inputs.use_jinja ? apply_jinja : non-jinja fallback). Without it the Qwen-Sharp template's thinking/image/tool formatting is silently dropped. Add USE_JINJA=1 -> --jinja. Experts: the RTX 3090 had ~5.4 GiB headroom; pull expert blocks 30,32,33,34 back to CUDA (each block ~630 MiB) from CPU for lower latency. Update the OVERRIDE_EXPERT regex, its test, and the KB with the headroom-check note.
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: ik_llama.cpp on Dankserver (Ornith Q6_K)
|
||||
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), speculative decoding disabled (3 parallel slots), and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
||||
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), speculative decoding disabled (3 parallel slots), the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
||||
tags:
|
||||
- ikllama
|
||||
- llamacpp
|
||||
@@ -144,17 +144,38 @@ cache RAM vs f16.
|
||||
> single-GPU host you must use `none` (or `layer`, which also needs >1).
|
||||
> Do not assume `--split-mode graph` degrades gracefully on one GPU.
|
||||
|
||||
### Expert offload (MoE on CPU)
|
||||
### Expert offload (MoE: CPU with a partial GPU slice)
|
||||
|
||||
The expert-offload pattern is unchanged from upstream llama.cpp: the
|
||||
The expert-offload pattern from upstream llama.cpp carries over: the
|
||||
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
|
||||
the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
|
||||
On 2026-08-26 we brought expert blocks **30, 32, 33, 34 back to CUDA**
|
||||
(~2.5 GiB) to use the ~5.4 GiB GPU headroom for lower latency; the rest of
|
||||
that CPU range stays on CPU.
|
||||
|
||||
- `OVERRIDE_EXPERT` — the layer-bounded regex (same as before the switch).
|
||||
- `OVERRIDE_EXPERT` — the layer-bounded regex listing only the blocks whose
|
||||
experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
|
||||
24|25|26|28|29)`. Blocks 30,32,33,34 (and the always-on full-attention +
|
||||
35-40 set) are left off the list so their experts load to CUDA.
|
||||
- `OVERRIDE_BUFT=CPU`.
|
||||
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
|
||||
integer (the literal `all` is rejected). 999 means "load as many tensor
|
||||
layers as possible", while the expert override still pins the exps to CPU.
|
||||
layers as possible", while the expert override still pins the listed exps
|
||||
to CPU.
|
||||
|
||||
To move more experts back to the GPU later, drop more block numbers from
|
||||
`OVERRIDE_EXPERT` (each block is ~630 MiB; verify `nvidia-smi` headroom
|
||||
first and keep ~2-3 GiB margin for KV/compute).
|
||||
|
||||
### Qwen-Sharp chat template via --jinja
|
||||
|
||||
`--chat-template-file` alone only stores the template text; ik_llama **does
|
||||
not** render a custom template as real Jinja unless `--jinja` is also given
|
||||
(`inputs.use_jinja ? common_chat_templates_apply_jinja : <non-jinja
|
||||
fallback>`). Without `--jinja` it warns it only supports "commonly used
|
||||
templates" and silently uses built-in handlers, dropping the Qwen-Sharp
|
||||
template's thinking / image / tool formatting. So the serve script passes
|
||||
both `--chat-template-file` and `--jinja` (gated on `USE_JINJA`, set to `1`).
|
||||
|
||||
### Speculative decoding: disabled (3 parallel slots)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user