Select hybrid expert placement from benchmark
This commit is contained in:
1 parent
339325719e
commit
f13124f977
6 files changed
+41
-31
No files matched your search
@@ -7,8 +7,9 @@
|
||||
# 3B-active MoE-dense hybrid with the MTP speculative head bundled in the GGUF.
|
||||
#
|
||||
# Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
|
||||
# --override-tensor pattern are pinned to CPU. The final output-adjacent
|
||||
# blocks stay on CUDA with all non-expert tensors, including the MTP head.
|
||||
# --override-tensor pattern are pinned to CPU. Selected full-attention and
|
||||
# final output-adjacent blocks stay on CUDA with all non-expert tensors,
|
||||
# including the MTP head.
|
||||
#
|
||||
# Spec: MoE model with a bundled MTP head -> native draft-mtp (the head is
|
||||
# auto-detected from the target model; b10121 creates the MTP draft context
|
||||
@@ -49,7 +50,7 @@ fi
|
||||
# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
|
||||
# the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
|
||||
# MoE expert weights and leave every other tensor on the GPU.
|
||||
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
|
||||
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
|
||||
: "${OVERRIDE_BUFT:=CPU}"
|
||||
: "${N_GPU_LAYERS:=all}"
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
|
||||
Reference in new issue
Block a user