frank-default-small: prefill-max ncmoe 35 / ubatch 3072
Prefill is the priority for this workload. Switch from decode-max (ncmoe 32 / ubatch 1024) to prefill-max: - n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s, ~1.2 GiB headroom at the 800k pool. - Gen is independent of ubatch; ub 3072 reaches best prefill with no decode cost. Docs synced (benchmark Chosen State, deployment, specs).
This commit is contained in:
1 parent
2b4e58a926
commit
01eb957f90
5 files changed
+28
-23
No files matched your search
@@ -32,10 +32,10 @@ fi
|
||||
: "${NGRAM_MOD_N_MATCH:=24}"
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${N_CPU_MOE:=32}" # 32 of 40 layers: MoE expert weights stay on CPU; 8 on GPU
|
||||
: "${N_CPU_MOE:=35}" # 35 of 40 layers: MoE expert weights stay on CPU; 5 on GPU
|
||||
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=1024}"
|
||||
: "${UBATCH_SIZE:=3072}"
|
||||
: "${CTX_CHECKPOINTS:=0}"
|
||||
: "${NO_MMAP:=0}"
|
||||
: "${TEMPERATURE:=1.0}"
|
||||
|
||||
Reference in new issue
Block a user