dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: pull expert blocks 28,29 back to GPU

Browse Source 
Fused MoE (fused_moe / fused_up_gate / fused_mmad) is on by default in
ik_llama - no flag needed; only -no-fmoe disables it. Slide two more
expert blocks (28,29) from CPU onto the RTX 3090 (~1.26 GiB) for the
fused-MoE run, leaving ~1.6 GiB headroom. Update override regex, test,
and KB (including a note that the headroom is now tighter than the
2-3 GiB ideal).
This commit is contained in:
Frank
2026-08-26 04:05:24 -07:00
parent 3f3b1bdb27
commit 23e905c712
3 changed files with 16 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+3 -2
View File
@@ -66,9 +66,10 @@ SPEC_TYPE=none
# CPU buft name on this host.
#
# 2026-08-26: brought back expert blocks 30,32,33,34 to CUDA (~2.5 GiB) to
# use the ~5.4 GiB GPU headroom for lower latency; the remaining larger-CPU
# use the ~5.4 GiB GPU headroom for lower latency; then 28,29 (~1.26 GiB)
# for the fused-MoE run. Left ~1.6 GiB headroom. The remaining larger-CPU
# range below those stays on CPU.
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
# ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to
# mean "load as many layers as possible". Expert exps are pinned to CPU by