frank-default-large: threads-batch=16 confirmed optimal + decode benchmark results
Decode sweep (TB=8,16,32, 8K+ prompt, 256 gen tokens): TB=8: 3.3 tok/s (76.6s) TB=16: 3.4 tok/s (74.8s) TB=32: 3.4 tok/s (75.0s) Decode is flat; TB=16 wins on prefill (5.1% faster) with zero decode tradeoff. Also: - Fix :: syntax error in serve-frank-large.sh - Document systemctl disable destroys unit file; use mask/unmask instead
This commit is contained in:
@@ -59,7 +59,7 @@ PARALLEL=2
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=4096
|
||||
NUM_THREADS=16
|
||||
THREAD_BATCH=32
|
||||
THREAD_BATCH=16
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
CTX_CHECKPOINTS=16
|
||||
|
||||
@@ -49,7 +49,7 @@ fi
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=4096}"
|
||||
: "${NUM_THREADS:=16}"
|
||||
: "${THREAD_BATCH:=32}"
|
||||
: "${THREAD_BATCH:=16}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${CTX_CHECKPOINTS:=16}"
|
||||
|
||||
Reference in New Issue
Block a user