frank-default-large: threads-batch=16 confirmed optimal + decode benchmark results
Decode sweep (TB=8,16,32, 8K+ prompt, 256 gen tokens): TB=8: 3.3 tok/s (76.6s) TB=16: 3.4 tok/s (74.8s) TB=32: 3.4 tok/s (75.0s) Decode is flat; TB=16 wins on prefill (5.1% faster) with zero decode tradeoff. Also: - Fix :: syntax error in serve-frank-large.sh - Document systemctl disable destroys unit file; use mask/unmask instead
This commit is contained in:
1 parent
26173df1fd
commit
cadc7f2768
4 files changed
+50
-6
No files matched your search
@@ -49,7 +49,7 @@ fi
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=4096}"
|
||||
: "${NUM_THREADS:=16}"
|
||||
: "${THREAD_BATCH:=32}"
|
||||
: "${THREAD_BATCH:=16}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${CTX_CHECKPOINTS:=16}"
|
||||
|
||||
Reference in new issue
Block a user