dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: pull expert blocks 28,29 back to GPU

Browse Source 
Fused MoE (fused_moe / fused_up_gate / fused_mmad) is on by default in
ik_llama - no flag needed; only -no-fmoe disables it. Slide two more
expert blocks (28,29) from CPU onto the RTX 3090 (~1.26 GiB) for the
fused-MoE run, leaving ~1.6 GiB headroom. Update override regex, test,
and KB (including a note that the headroom is now tighter than the
2-3 GiB ideal).
This commit is contained in:
Frank
2026-08-26 04:05:24 -07:00
parent 3f3b1bdb27
commit 23e905c712
3 changed files with 16 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_small_config.bats
+1 -1
View File
@@ -22,7 +22,7 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
@test "frank-default-small keeps full-attention and final blocks on CUDA" {
source "$ENV_FILE"
[[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
[[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
[[ "$OVERRIDE_EXPERT" != *'|3|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|7|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|11|'* ]]