perf: frank-default-small batch 3072 / ubatch 1536 + ctx-checkpoints

Bigger logical/physical batches for prefill throughput. Enable
--ctx-checkpoints to park attention checkpoints in RAM, holding peak GPU
memory down so the larger ubatch fits the 24GB VRAM alongside 3x262144 KV
without OOM. Weight split unchanged (experts CPU, non-experts GPU).
This commit is contained in:
Frank committed 2026-08-11 01:33:28 -07:00
1 parent 35c44c4123
commit e45f457ad4
2 files changed
+12

No files matched your search

+6
View File
@@ -32,6 +32,9 @@ fi
: "${CACHE_TYPE_V:=f16}"
: "${N_CPU_MOE:=40}" # all 40 layers: MoE expert weights stay on CPU
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${CTX_CHECKPOINTS:=0}"
: "${THREADS:=8}" # 7700X physical cores
: "${THREADS_HTTP:=8}"
@@ -60,6 +63,9 @@ exec "$LLAMA_SERVER" \
--parallel "$PARALLEL" \
--n-cpu-moe "$N_CPU_MOE" \
--gpu-layers "$N_GPU_LAYERS" \
--batch-size "$BATCH_SIZE" \
--ubatch-size "$UBATCH_SIZE" \
--ctx-checkpoints "$CTX_CHECKPOINTS" \
--threads "$THREADS" \
--threads-http "$THREADS_HTTP" \
--alias "$ALIAS" \