---
name: bench-llama-server-params
description: "Benchmark llama.cpp server-side parameters (--threads-batch, --ubatch-size, --spec-type) using HTTP API with max_tokens=1 for prefill, max_tokens=16384+ for decode"
---

# Benchmarking llama.cpp Server-Side Parameters

## Problem
`llama-bench` benchmarks the bench binary itself, not the running server.
It cannot measure the effect of server-side parameters like `--threads-batch`,
`--ubatch-size`, `--spec-type`, or `--parallel`.

## Solution: HTTP-Based Benchmarking

### Prefill throughput (prompt processing)
1. Start the server with the target parameter value
2. Send a large prompt via the chat API with `max_tokens=1`
3. Measure `prompt_tokens / wall_time` from the response usage

`max_tokens=1` is correct here — you want only prompt processing time,
not generation. The server crashes on `max_tokens=0`.

### Decode throughput (generation)
1. Start the server with the target parameter value
2. Send a prompt that generates a long response
3. Use `max_tokens=16384` minimum (16k) — **never `max_tokens=1`** for decode
   benchmarking. One token tells you nothing about sustained generation rate.
4. Measure `predicted_per_second` from the response `timings` object, or
   compute `completion_tokens / wall_time`

### Why at least 16k tokens for decode
- Decode throughput varies over the generation (warmup, cache growth, MoE
  routing). A single token is noise.
- Speculative decoding acceptance rates stabilize over thousands of tokens.
- Short generations finish before the draft head reaches steady state.
- 16k tokens gives a stable average at ~22 tok/s (≈12 min per run).

### Key Pattern
```python
# Decode benchmark — measure generation throughput
body = json.dumps({
    "model": "<alias>",
    "messages": [{"role": "user", "content": "<prompt that generates 10k+ tokens>"}],
    "max_tokens": 16384,   # 16k minimum for decode; use max_tokens=1 ONLY for prefill
    "temperature": 0,     # deterministic for reproducibility
    "stream": False
})
# Measure wall time from request send to response receipt
# Get predicted_per_second from timings, or completion_tokens / wall_time
```

### When to Use
- `--threads-batch` (prefill): use `max_tokens=1`
- `--ubatch-size` (decode): use `max_tokens=16384+`
- `--spec-type` / speculative decoding: use `max_tokens=16384+` (decode only)
- Any server-side parameter that `llama-bench` cannot target

### Benchmarking policy (MANDATORY)
1. **STOP the production service** before running any benchmark
2. **Mask** the service (`systemctl mask`, never `disable` — on these hosts
   `disable` deletes the unit file from `/etc/systemd/system/`)
3. Run the bench
4. **Unmask + restart** the service when done
5. **Verify health** before reporting results

### Example Scripts
- `scripts/bench-threads-batch-http.py` — prefill sweep (max_tokens=1)
- `scripts/bench-decode-threads-batch.py` — decode sweep (max_tokens=256+)
- `scripts/bench-spec-decode.py` — speculative decoding config sweep