frank-default-large: threads-batch=16 confirmed optimal + decode benchmark results
Decode sweep (TB=8,16,32, 8K+ prompt, 256 gen tokens): TB=8: 3.3 tok/s (76.6s) TB=16: 3.4 tok/s (74.8s) TB=32: 3.4 tok/s (75.0s) Decode is flat; TB=16 wins on prefill (5.1% faster) with zero decode tradeoff. Also: - Fix :: syntax error in serve-frank-large.sh - Document systemctl disable destroys unit file; use mask/unmask instead
This commit is contained in:
1 parent
26173df1fd
commit
cadc7f2768
4 files changed
+50
-6
No files matched your search
@@ -109,6 +109,41 @@ sized to 85% of context window:
|
||||
|
||||
Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s
|
||||
generation (median of 3 runs).
|
||||
## Benchmark: threads-batch tuning
|
||||
|
||||
Sweeped `--threads-batch` (16, 32, 48, 64, 80, 96, 128) with a 64K-token
|
||||
prompt via HTTP API (`max_tokens=1`). Fresh server start per value.
|
||||
|
||||
| threads-batch | prefill tok/s | wall time |
|
||||
|---------------|--------------|-----------|
|
||||
| **16** | **373.5** | 62.9s |
|
||||
| 32 | 354.4 | 66.3s |
|
||||
| 48 | 344.1 | 68.3s |
|
||||
| 64 | 344.7 | 68.2s |
|
||||
| 80 | 344.4 | 68.2s |
|
||||
| 96 | 357.1 | 65.8s |
|
||||
| 128 | 321.1 | 73.2s |
|
||||
|
||||
**Optimal: `--threads-batch 16`** (373.5 tok/s). Lower thread batching wins
|
||||
for prefill on this model -- fewer threads compete for GPU resources during
|
||||
the prompt processing phase. The default of 32 is 5.1% slower. Above 64,
|
||||
performance plateaus then declines (128 is 14% worse than optimal).
|
||||
|
||||
## Benchmark: decode throughput
|
||||
|
||||
Sweeped `--threads-batch` (8, 16, 32) with a ~23.5K-token code prompt and
|
||||
256 generated tokens via HTTP API. Fresh server start per value.
|
||||
|
||||
| threads-batch | gen tokens | wall time | gen tok/s |
|
||||
|---------------|-----------|-----------|-----------|
|
||||
| 16 | 256 | 74.8s | 3.4 |
|
||||
| 32 | 256 | 75.0s | 3.4 |
|
||||
| 8 | 256 | 76.6s | 3.3 |
|
||||
|
||||
**Decode throughput is essentially flat across all tested values.** TB=16 and
|
||||
TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s.
|
||||
Combined with the 5.1% prefill advantage at lower thread counts, `--threads-batch
|
||||
16` is the optimal setting for both phases.
|
||||
|
||||
## Serving Profile
|
||||
|
||||
@@ -126,7 +161,7 @@ llama-server \
|
||||
--ctx-size 524288 --parallel 2 \
|
||||
--batch-size 4096 --ubatch-size 4096 \
|
||||
--cache-type-k f16 --cache-type-v f16 \
|
||||
--threads 16 --threads-batch 32 \
|
||||
--threads 16 --threads-batch 16 \
|
||||
--ctx-checkpoints 16 --cache-ram 8192 \
|
||||
--cache-prompt --cache-idle-slots --metrics \
|
||||
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
|
||||
|
||||
Reference in new issue
Block a user