frank-default-small: pull expert blocks 28,29 back to GPU
Fused MoE (fused_moe / fused_up_gate / fused_mmad) is on by default in ik_llama - no flag needed; only -no-fmoe disables it. Slide two more expert blocks (28,29) from CPU onto the RTX 3090 (~1.26 GiB) for the fused-MoE run, leaving ~1.6 GiB headroom. Update override regex, test, and KB (including a note that the headroom is now tighter than the 2-3 GiB ideal).
This commit is contained in:
@@ -149,13 +149,13 @@ cache RAM vs f16.
|
||||
The expert-offload pattern from upstream llama.cpp carries over: the
|
||||
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
|
||||
the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
|
||||
On 2026-08-26 we brought expert blocks **30, 32, 33, 34 back to CUDA**
|
||||
(~2.5 GiB) to use the ~5.4 GiB GPU headroom for lower latency; the rest of
|
||||
that CPU range stays on CPU.
|
||||
On 2026-08-26 we brought expert blocks **30, 32, 33, 34** back to CUDA
|
||||
(~2.5 GiB), then **28 and 29** (~1.26 GiB) for the fused-MoE run, to use the
|
||||
GPU headroom for lower latency; the rest of that CPU range stays on CPU.
|
||||
|
||||
- `OVERRIDE_EXPERT` — the layer-bounded regex listing only the blocks whose
|
||||
experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
|
||||
24|25|26|28|29)`. Blocks 30,32,33,34 (and the always-on full-attention +
|
||||
24|25|26)`. Blocks 28,29,30,32,33,34 (and the always-on full-attention +
|
||||
35-40 set) are left off the list so their experts load to CUDA.
|
||||
- `OVERRIDE_BUFT=CPU`.
|
||||
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
|
||||
@@ -163,9 +163,16 @@ that CPU range stays on CPU.
|
||||
layers as possible", while the expert override still pins the listed exps
|
||||
to CPU.
|
||||
|
||||
> **Fused MoE** (`fused_moe`, `fused_up_gate`, `fused_mmad`) is **on by
|
||||
> default** in ik_llama — there is no positive flag; only `-no-fmoe`
|
||||
> disables it. `only_active_experts` is also on by default. Confirmed in the
|
||||
> boot log for the running service.
|
||||
|
||||
To move more experts back to the GPU later, drop more block numbers from
|
||||
`OVERRIDE_EXPERT` (each block is ~630 MiB; verify `nvidia-smi` headroom
|
||||
first and keep ~2-3 GiB margin for KV/compute).
|
||||
first and keep ~2-3 GiB margin for KV/compute). After 28,29 were added the
|
||||
free headroom was ~1.6 GiB, which is tighter than ideal — add more only
|
||||
with a headroom check.
|
||||
|
||||
### Qwen-Sharp chat template via --jinja
|
||||
|
||||
|
||||
Reference in New Issue
Block a user