frank-default-small: pull expert blocks 28,29 back to GPU

Fused MoE (fused_moe / fused_up_gate / fused_mmad) is on by default in
ik_llama - no flag needed; only -no-fmoe disables it. Slide two more
expert blocks (28,29) from CPU onto the RTX 3090 (~1.26 GiB) for the
fused-MoE run, leaving ~1.6 GiB headroom. Update override regex, test,
and KB (including a note that the headroom is now tighter than the
2-3 GiB ideal).
This commit is contained in:
Frank
2026-08-26 04:05:24 -07:00
parent 3f3b1bdb27
commit 23e905c712
3 changed files with 16 additions and 8 deletions
+12 -5
View File
@@ -149,13 +149,13 @@ cache RAM vs f16.
The expert-offload pattern from upstream llama.cpp carries over: the
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
On 2026-08-26 we brought expert blocks **30, 32, 33, 34 back to CUDA**
(~2.5 GiB) to use the ~5.4 GiB GPU headroom for lower latency; the rest of
that CPU range stays on CPU.
On 2026-08-26 we brought expert blocks **30, 32, 33, 34** back to CUDA
(~2.5 GiB), then **28 and 29** (~1.26 GiB) for the fused-MoE run, to use the
GPU headroom for lower latency; the rest of that CPU range stays on CPU.
- `OVERRIDE_EXPERT` — the layer-bounded regex listing only the blocks whose
experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
24|25|26|28|29)`. Blocks 30,32,33,34 (and the always-on full-attention +
24|25|26)`. Blocks 28,29,30,32,33,34 (and the always-on full-attention +
35-40 set) are left off the list so their experts load to CUDA.
- `OVERRIDE_BUFT=CPU`.
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
@@ -163,9 +163,16 @@ that CPU range stays on CPU.
layers as possible", while the expert override still pins the listed exps
to CPU.
> **Fused MoE** (`fused_moe`, `fused_up_gate`, `fused_mmad`) is **on by
> default** in ik_llama — there is no positive flag; only `-no-fmoe`
> disables it. `only_active_experts` is also on by default. Confirmed in the
> boot log for the running service.
To move more experts back to the GPU later, drop more block numbers from
`OVERRIDE_EXPERT` (each block is ~630 MiB; verify `nvidia-smi` headroom
first and keep ~2-3 GiB margin for KV/compute).
first and keep ~2-3 GiB margin for KV/compute). After 28,29 were added the
free headroom was ~1.6 GiB, which is tighter than ideal — add more only
with a headroom check.
### Qwen-Sharp chat template via --jinja