Select hybrid expert placement from benchmark

This commit is contained in:
Frank committed 2026-08-25 01:08:00 -07:00
1 parent 339325719e
commit f13124f977
6 files changed
+41 -31

No files matched your search

+4 -3
View File
@@ -7,8 +7,9 @@
# 3B-active MoE-dense hybrid with the MTP speculative head bundled in the GGUF.
#
# Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
# --override-tensor pattern are pinned to CPU. The final output-adjacent
# blocks stay on CUDA with all non-expert tensors, including the MTP head.
# --override-tensor pattern are pinned to CPU. Selected full-attention and
# final output-adjacent blocks stay on CUDA with all non-expert tensors,
# including the MTP head.
#
# Spec: MoE model with a bundled MTP head -> native draft-mtp (the head is
# auto-detected from the target model; b10121 creates the MTP draft context
@@ -49,7 +50,7 @@ fi
# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
# the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
# MoE expert weights and leave every other tensor on the GPU.
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
: "${OVERRIDE_BUFT:=CPU}"
: "${N_GPU_LAYERS:=all}"
: "${BATCH_SIZE:=4096}"