dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Select hybrid expert placement from benchmark

Browse Source
This commit is contained in:
Frank committed 2026-08-25 01:08:00 -07:00
1 parent 339325719e
commit f13124f977
6 files changed
+41 -31
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-default-small.env
+6 -5
View File
@@ -42,11 +42,12 @@ SPEC_DRAFT_N_MIN=0
SPEC_DRAFT_P_MIN=0.0
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep the four
# output-adjacent blocks 33-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-32 to CPU. All non-expert tensors stay on
# the 24 GB card. Buffer type "CPU" is the CPU buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
# layer audit found 630 MiB of expert tensors per block. Keep the selected
# full-attention blocks 27 and 31 plus output-adjacent blocks 33-40 on CUDA,
# including the bundled MTP block 40. Move all other expert tensors to CPU.
# All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the CPU
# buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
N_GPU_LAYERS=all