dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Select hybrid expert placement from benchmark

Browse Source
This commit is contained in:
Frank
2026-08-25 01:08:00 -07:00
parent 339325719e
commit f13124f977
6 changed files with 41 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+6 -5
View File
@@ -42,11 +42,12 @@ SPEC_DRAFT_N_MIN=0
SPEC_DRAFT_P_MIN=0.0
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep the four
# output-adjacent blocks 33-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-32 to CPU. All non-expert tensors stay on
# the 24 GB card. Buffer type "CPU" is the CPU buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
# layer audit found 630 MiB of expert tensors per block. Keep the selected
# full-attention blocks 27 and 31 plus output-adjacent blocks 33-40 on CUDA,
# including the bundled MTP block 40. Move all other expert tensors to CPU.
# All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the CPU
# buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
N_GPU_LAYERS=all
scripts/serve-frank-small.sh
+4 -3
View File
@@ -7,8 +7,9 @@
# 3B-active MoE-dense hybrid with the MTP speculative head bundled in the GGUF.
#
# Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
# --override-tensor pattern are pinned to CPU. The final output-adjacent
# blocks stay on CUDA with all non-expert tensors, including the MTP head.
# --override-tensor pattern are pinned to CPU. Selected full-attention and
# final output-adjacent blocks stay on CUDA with all non-expert tensors,
# including the MTP head.
#
# Spec: MoE model with a bundled MTP head -> native draft-mtp (the head is
# auto-detected from the target model; b10121 creates the MTP draft context
@@ -49,7 +50,7 @@ fi
# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
# the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
# MoE expert weights and leave every other tensor on the GPU.
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
: "${OVERRIDE_BUFT:=CPU}"
: "${N_GPU_LAYERS:=all}"
: "${BATCH_SIZE:=4096}"