perf: frank-default-small ctx 200000/stream, --no-mmap, batch 3072/1536
Lower each stream to 200000 ctx (pool 600000) to free KV VRAM for the larger batch. Add --no-mmap so the CPU-resident MoE experts load eagerly into RAM (avoids decode page faults). Keep KV on GPU. Experts already fully on CPU.
This commit is contained in:
1 parent
e45f457ad4
commit
9971f83e8b
2 files changed
+8
-2
No files matched your search
@@ -35,6 +35,7 @@ fi
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${CTX_CHECKPOINTS:=0}"
|
||||
: "${NO_MMAP:=0}"
|
||||
: "${THREADS:=8}" # 7700X physical cores
|
||||
: "${THREADS_HTTP:=8}"
|
||||
|
||||
@@ -66,6 +67,7 @@ exec "$LLAMA_SERVER" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--ctx-checkpoints "$CTX_CHECKPOINTS" \
|
||||
$([ "$NO_MMAP" = "1" ] && printf -- "--no-mmap") \
|
||||
--threads "$THREADS" \
|
||||
--threads-http "$THREADS_HTTP" \
|
||||
--alias "$ALIAS" \
|
||||
|
||||
Reference in new issue
Block a user