From cadc7f2768f06edb5ac2e4722405a0579548f7c6 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 27 Aug 2026 13:20:52 -0700 Subject: [PATCH] frank-default-large: threads-batch=16 confirmed optimal + decode benchmark results Decode sweep (TB=8,16,32, 8K+ prompt, 256 gen tokens): TB=8: 3.3 tok/s (76.6s) TB=16: 3.4 tok/s (74.8s) TB=32: 3.4 tok/s (75.0s) Decode is flat; TB=16 wins on prefill (5.1% faster) with zero decode tradeoff. Also: - Fix :: syntax error in serve-frank-large.sh - Document systemctl disable destroys unit file; use mask/unmask instead --- AGENTS.md | 15 +++++++++--- docs/LLM/kingjones-strix-rocmfp4.md | 37 ++++++++++++++++++++++++++++- scripts/frank-default-large.env | 2 +- scripts/serve-frank-large.sh | 2 +- 4 files changed, 50 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a887949..23497d1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -227,14 +227,23 @@ ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes. DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE. 1. Stop the service before any benchmark: - `sudo systemctl stop frank-default-small` (Dankserver) or - `sudo systemctl stop frank-default-large` (Dankclaw). + `sudo systemctl stop frank-default-large` (Dankclaw) or + `sudo systemctl stop frank-default-small` (Dankserver). + Then mask it to prevent the self-healing unit from restarting: + `sudo systemctl mask frank-default-large` (Dankclaw). + **CRITICAL WARNING:** Do NOT use `systemctl disable` on these service units. + On these hosts, `systemctl disable` doesn't just remove symlinks — it **deletes + the unit file itself** from `/etc/systemd/system/`. You'd have to manually + copy the unit file back from this repo, run `daemon-reload`, then enable and + start. Mask (`systemctl mask`) creates a symlink to `/dev/null` instead, which + is safe and reversible. 2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM). Scratch ports are not used for tests or benches on the LLM hosts — a test process cannot share the single GPU with the running service anyway (the model + KV fill the card), so changes are validated directly against the service itself. -3. Restart the service when the bench completes: `sudo systemctl start `. +3. Restart the service when the bench completes: unmask, then start: + `sudo systemctl unmask frank-default-large && sudo systemctl start frank-default-large` (Dankclaw). 4. Verify health before reporting: `curl http://192.168.0.235:7770/health` or `curl http://192.168.0.143:7770/health`. diff --git a/docs/LLM/kingjones-strix-rocmfp4.md b/docs/LLM/kingjones-strix-rocmfp4.md index 63aca40..902c221 100644 --- a/docs/LLM/kingjones-strix-rocmfp4.md +++ b/docs/LLM/kingjones-strix-rocmfp4.md @@ -109,6 +109,41 @@ sized to 85% of context window: Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s generation (median of 3 runs). +## Benchmark: threads-batch tuning + +Sweeped `--threads-batch` (16, 32, 48, 64, 80, 96, 128) with a 64K-token +prompt via HTTP API (`max_tokens=1`). Fresh server start per value. + +| threads-batch | prefill tok/s | wall time | +|---------------|--------------|-----------| +| **16** | **373.5** | 62.9s | +| 32 | 354.4 | 66.3s | +| 48 | 344.1 | 68.3s | +| 64 | 344.7 | 68.2s | +| 80 | 344.4 | 68.2s | +| 96 | 357.1 | 65.8s | +| 128 | 321.1 | 73.2s | + +**Optimal: `--threads-batch 16`** (373.5 tok/s). Lower thread batching wins +for prefill on this model -- fewer threads compete for GPU resources during +the prompt processing phase. The default of 32 is 5.1% slower. Above 64, +performance plateaus then declines (128 is 14% worse than optimal). + +## Benchmark: decode throughput + +Sweeped `--threads-batch` (8, 16, 32) with a ~23.5K-token code prompt and +256 generated tokens via HTTP API. Fresh server start per value. + +| threads-batch | gen tokens | wall time | gen tok/s | +|---------------|-----------|-----------|-----------| +| 16 | 256 | 74.8s | 3.4 | +| 32 | 256 | 75.0s | 3.4 | +| 8 | 256 | 76.6s | 3.3 | + +**Decode throughput is essentially flat across all tested values.** TB=16 and +TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s. +Combined with the 5.1% prefill advantage at lower thread counts, `--threads-batch +16` is the optimal setting for both phases. ## Serving Profile @@ -126,7 +161,7 @@ llama-server \ --ctx-size 524288 --parallel 2 \ --batch-size 4096 --ubatch-size 4096 \ --cache-type-k f16 --cache-type-v f16 \ - --threads 16 --threads-batch 32 \ + --threads 16 --threads-batch 16 \ --ctx-checkpoints 16 --cache-ram 8192 \ --cache-prompt --cache-idle-slots --metrics \ --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \ diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 2bf121f..60c0fd8 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -59,7 +59,7 @@ PARALLEL=2 BATCH_SIZE=4096 UBATCH_SIZE=4096 NUM_THREADS=16 -THREAD_BATCH=32 +THREAD_BATCH=16 CACHE_TYPE_K=f16 CACHE_TYPE_V=f16 CTX_CHECKPOINTS=16 diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 5c65cb7..9c7747d 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -49,7 +49,7 @@ fi : "${BATCH_SIZE:=4096}" : "${UBATCH_SIZE:=4096}" : "${NUM_THREADS:=16}" -: "${THREAD_BATCH:=32}" +: "${THREAD_BATCH:=16}" : "${CACHE_TYPE_K:=f16}" : "${CACHE_TYPE_V:=f16}" : "${CTX_CHECKPOINTS:=16}"