70 lines
2.9 KiB
Markdown
70 lines
2.9 KiB
Markdown
---
|
|
name: bench-llama-server-params
|
|
description: "Benchmark llama.cpp server-side parameters (--threads-batch, --ubatch-size, --spec-type) using HTTP API with max_tokens=1 for prefill, max_tokens=16384+ for decode"
|
|
---
|
|
|
|
# Benchmarking llama.cpp Server-Side Parameters
|
|
|
|
## Problem
|
|
`llama-bench` benchmarks the bench binary itself, not the running server.
|
|
It cannot measure the effect of server-side parameters like `--threads-batch`,
|
|
`--ubatch-size`, `--spec-type`, or `--parallel`.
|
|
|
|
## Solution: HTTP-Based Benchmarking
|
|
|
|
### Prefill throughput (prompt processing)
|
|
1. Start the server with the target parameter value
|
|
2. Send a large prompt via the chat API with `max_tokens=1`
|
|
3. Measure `prompt_tokens / wall_time` from the response usage
|
|
|
|
`max_tokens=1` is correct here — you want only prompt processing time,
|
|
not generation. The server crashes on `max_tokens=0`.
|
|
|
|
### Decode throughput (generation)
|
|
1. Start the server with the target parameter value
|
|
2. Send a prompt that generates a long response
|
|
3. Use `max_tokens=16384` minimum (16k) — **never `max_tokens=1`** for decode
|
|
benchmarking. One token tells you nothing about sustained generation rate.
|
|
4. Measure `predicted_per_second` from the response `timings` object, or
|
|
compute `completion_tokens / wall_time`
|
|
|
|
### Why at least 16k tokens for decode
|
|
- Decode throughput varies over the generation (warmup, cache growth, MoE
|
|
routing). A single token is noise.
|
|
- Speculative decoding acceptance rates stabilize over thousands of tokens.
|
|
- Short generations finish before the draft head reaches steady state.
|
|
- 16k tokens gives a stable average at ~22 tok/s (≈12 min per run).
|
|
|
|
### Key Pattern
|
|
```python
|
|
# Decode benchmark — measure generation throughput
|
|
body = json.dumps({
|
|
"model": "<alias>",
|
|
"messages": [{"role": "user", "content": "<prompt that generates 10k+ tokens>"}],
|
|
"max_tokens": 16384, # 16k minimum for decode; use max_tokens=1 ONLY for prefill
|
|
"temperature": 0, # deterministic for reproducibility
|
|
"stream": False
|
|
})
|
|
# Measure wall time from request send to response receipt
|
|
# Get predicted_per_second from timings, or completion_tokens / wall_time
|
|
```
|
|
|
|
### When to Use
|
|
- `--threads-batch` (prefill): use `max_tokens=1`
|
|
- `--ubatch-size` (decode): use `max_tokens=16384+`
|
|
- `--spec-type` / speculative decoding: use `max_tokens=16384+` (decode only)
|
|
- Any server-side parameter that `llama-bench` cannot target
|
|
|
|
### Benchmarking policy (MANDATORY)
|
|
1. **STOP the production service** before running any benchmark
|
|
2. **Mask** the service (`systemctl mask`, never `disable` — on these hosts
|
|
`disable` deletes the unit file from `/etc/systemd/system/`)
|
|
3. Run the bench
|
|
4. **Unmask + restart** the service when done
|
|
5. **Verify health** before reporting results
|
|
|
|
### Example Scripts
|
|
- `scripts/bench-threads-batch-http.py` — prefill sweep (max_tokens=1)
|
|
- `scripts/bench-decode-threads-batch.py` — decode sweep (max_tokens=256+)
|
|
- `scripts/bench-spec-decode.py` — speculative decoding config sweep
|