Set equal batch and ubatch at 8k context
This commit is contained in:
@@ -23,8 +23,8 @@ for auditability only. Their results are invalid and must not be used.
|
||||
## Sweep
|
||||
|
||||
The requested points are `256, 512, 768, 1024, 1280, 1536, 1792, 2048,
|
||||
3072, 4096` at 32K context. Each point runs three 128-token repetitions by
|
||||
default.
|
||||
3072, 4096` at 8K context, with `batch` equal to `ubatch` at every point.
|
||||
Each point runs three 128-token repetitions by default.
|
||||
|
||||
Run on Dankclaw from the repository checkout:
|
||||
|
||||
@@ -34,8 +34,8 @@ sudo systemctl stop frank-default-large
|
||||
sudo systemctl start frank-default-large
|
||||
```
|
||||
|
||||
The script uses the active Q6 AGENT model, a fixed 32K prompt workload, and
|
||||
`llama-bench` JSONL output. The ubatch-only sweep is:
|
||||
The script uses the active Q6 AGENT model, an 8K prompt workload, equal
|
||||
`batch` and `ubatch` values, and `llama-bench` JSONL output. The sweep is:
|
||||
```text
|
||||
256, 512, 768, 1024, 1280, 1536, 1792, 2048, 3072, 4096
|
||||
```
|
||||
|
||||
@@ -2,9 +2,8 @@
|
||||
# Benchmark ubatch with llama-bench for frank-default-large.
|
||||
#
|
||||
# This is intentionally a ubatch-only benchmark. It uses one llama-bench
|
||||
# process, a fixed 32K prompt workload, and a fixed scheduler size large
|
||||
# enough to hold that workload. No server process, HTTP requests, or log
|
||||
# scraping are used.
|
||||
# process, an 8K prompt workload, and equal batch/ubatch values. No server
|
||||
# process, HTTP requests, or log scraping are used.
|
||||
#
|
||||
# Run on Dankclaw after stopping frank-default-large:
|
||||
# sudo systemctl stop frank-default-large
|
||||
@@ -15,7 +14,7 @@ set -euo pipefail
|
||||
: "${LLAMA_BENCH:=/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-bench}"
|
||||
: "${MODEL:=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf}"
|
||||
: "${UBATCH_VALUES:=256,512,768,1024,1280,1536,1792,2048,3072,4096}"
|
||||
: "${CONTEXT:=32768}"
|
||||
: "${CONTEXT:=8192}"
|
||||
: "${GEN_TOKENS:=128}"
|
||||
: "${REPETITIONS:=3}"
|
||||
: "${DEVICE:=ROCm0}"
|
||||
@@ -34,7 +33,7 @@ exec "$LLAMA_BENCH" \
|
||||
--model "$MODEL" \
|
||||
--n-prompt "$CONTEXT" \
|
||||
--n-gen "$GEN_TOKENS" \
|
||||
--batch-size "$CONTEXT" \
|
||||
--batch-size "$UBATCH_VALUES" \
|
||||
--ubatch-size "$UBATCH_VALUES" \
|
||||
--device "$DEVICE" \
|
||||
--n-gpu-layers 999 \
|
||||
|
||||
Reference in New Issue
Block a user