diff --git a/docs/LLM/ikllama-ornith-dankserver.md b/docs/LLM/ikllama-ornith-dankserver.md index bca6a3c..c383a25 100644 --- a/docs/LLM/ikllama-ornith-dankserver.md +++ b/docs/LLM/ikllama-ornith-dankserver.md @@ -149,13 +149,13 @@ cache RAM vs f16. The expert-offload pattern from upstream llama.cpp carries over: the `--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA. -On 2026-08-26 we brought expert blocks **30, 32, 33, 34 back to CUDA** -(~2.5 GiB) to use the ~5.4 GiB GPU headroom for lower latency; the rest of -that CPU range stays on CPU. +On 2026-08-26 we brought expert blocks **30, 32, 33, 34** back to CUDA +(~2.5 GiB), then **28 and 29** (~1.26 GiB) for the fused-MoE run, to use the +GPU headroom for lower latency; the rest of that CPU range stays on CPU. - `OVERRIDE_EXPERT` — the layer-bounded regex listing only the blocks whose experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22| - 24|25|26|28|29)`. Blocks 30,32,33,34 (and the always-on full-attention + + 24|25|26)`. Blocks 28,29,30,32,33,34 (and the always-on full-attention + 35-40 set) are left off the list so their experts load to CUDA. - `OVERRIDE_BUFT=CPU`. - `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an @@ -163,9 +163,16 @@ that CPU range stays on CPU. layers as possible", while the expert override still pins the listed exps to CPU. +> **Fused MoE** (`fused_moe`, `fused_up_gate`, `fused_mmad`) is **on by +> default** in ik_llama — there is no positive flag; only `-no-fmoe` +> disables it. `only_active_experts` is also on by default. Confirmed in the +> boot log for the running service. + To move more experts back to the GPU later, drop more block numbers from `OVERRIDE_EXPERT` (each block is ~630 MiB; verify `nvidia-smi` headroom -first and keep ~2-3 GiB margin for KV/compute). +first and keep ~2-3 GiB margin for KV/compute). After 28,29 were added the +free headroom was ~1.6 GiB, which is tighter than ideal — add more only +with a headroom check. ### Qwen-Sharp chat template via --jinja diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index c5dba45..7a65565 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -66,9 +66,10 @@ SPEC_TYPE=none # CPU buft name on this host. # # 2026-08-26: brought back expert blocks 30,32,33,34 to CUDA (~2.5 GiB) to -# use the ~5.4 GiB GPU headroom for lower latency; the remaining larger-CPU +# use the ~5.4 GiB GPU headroom for lower latency; then 28,29 (~1.26 GiB) +# for the fused-MoE run. Left ~1.6 GiB headroom. The remaining larger-CPU # range below those stays on CPU. -OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29)\.ffn_(up|down|gate|gate_up)_(ch|)exps' +OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_BUFT=CPU # ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to # mean "load as many layers as possible". Expert exps are pinned to CPU by diff --git a/tests/frank_small_config.bats b/tests/frank_small_config.bats index 18577a0..59056d4 100644 --- a/tests/frank_small_config.bats +++ b/tests/frank_small_config.bats @@ -22,7 +22,7 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" @test "frank-default-small keeps full-attention and final blocks on CUDA" { source "$ENV_FILE" - [[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]] + [[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]] [[ "$OVERRIDE_EXPERT" != *'|3|'* ]] [[ "$OVERRIDE_EXPERT" != *'|7|'* ]] [[ "$OVERRIDE_EXPERT" != *'|11|'* ]]