frank-default-small: prefill-max ncmoe 35 / ubatch 3072

Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
  ~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
  decode cost. Docs synced (benchmark Chosen State, deployment, specs).
This commit is contained in:
Frank committed 2026-08-13 23:07:30 -07:00
1 parent 2b4e58a926
commit 01eb957f90
5 files changed
+28 -23

No files matched your search

+2 -2
View File
@@ -32,10 +32,10 @@ fi
: "${NGRAM_MOD_N_MATCH:=24}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=f16}"
: "${N_CPU_MOE:=32}" # 32 of 40 layers: MoE expert weights stay on CPU; 8 on GPU
: "${N_CPU_MOE:=35}" # 35 of 40 layers: MoE expert weights stay on CPU; 5 on GPU
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=1024}"
: "${UBATCH_SIZE:=3072}"
: "${CTX_CHECKPOINTS:=0}"
: "${NO_MMAP:=0}"
: "${TEMPERATURE:=1.0}"