frank-default-large: threads-batch=16 confirmed optimal + decode benchmark results

Decode sweep (TB=8,16,32, 8K+ prompt, 256 gen tokens):
  TB=8:  3.3 tok/s (76.6s)
  TB=16: 3.4 tok/s (74.8s)
  TB=32: 3.4 tok/s (75.0s)

Decode is flat; TB=16 wins on prefill (5.1% faster) with zero decode tradeoff.

Also:
- Fix :: syntax error in serve-frank-large.sh
- Document systemctl disable destroys unit file; use mask/unmask instead
This commit is contained in:
Frank
2026-08-27 13:20:52 -07:00
parent 26173df1fd
commit cadc7f2768
4 changed files with 50 additions and 6 deletions
+1 -1
View File
@@ -59,7 +59,7 @@ PARALLEL=2
BATCH_SIZE=4096
UBATCH_SIZE=4096
NUM_THREADS=16
THREAD_BATCH=32
THREAD_BATCH=16
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
CTX_CHECKPOINTS=16
+1 -1
View File
@@ -49,7 +49,7 @@ fi
: "${BATCH_SIZE:=4096}"
: "${UBATCH_SIZE:=4096}"
: "${NUM_THREADS:=16}"
: "${THREAD_BATCH:=32}"
: "${THREAD_BATCH:=16}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
: "${CTX_CHECKPOINTS:=16}"