dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Tune small model expert placement

Browse Source
This commit is contained in:
Frank
2026-08-25 00:31:05 -07:00
parent af6ffeeb64
commit 7a17984dbc
5 changed files with 16 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+3 -3
View File
@@ -43,10 +43,10 @@ SPEC_DRAFT_P_MIN=0.0
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep the four
# output-adjacent blocks 37-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-36 to CPU. All non-expert tensors stay on
# output-adjacent blocks 36-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-35 to CPU. All non-expert tensors stay on
# the 24 GB card. Buffer type "CPU" is the CPU buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
N_GPU_LAYERS=all