command-center/agent/managed-skills/bench-llama-server-params/SKILL.md

name, description

name	description
bench-llama-server-params	Benchmark llama.cpp server-side parameters (--threads-batch, --ubatch-size, --spec-type) using HTTP API with max_tokens=1 for prefill, max_tokens=16384+ for decode

Benchmarking llama.cpp Server-Side Parameters

Problem

llama-bench benchmarks the bench binary itself, not the running server. It cannot measure the effect of server-side parameters like --threads-batch, --ubatch-size, --spec-type, or --parallel.

Solution: HTTP-Based Benchmarking

Prefill throughput (prompt processing)

1. Start the server with the target parameter value
2. Send a large prompt via the chat API with max_tokens=1
3. Measure prompt_tokens / wall_time from the response usage

max_tokens=1 is correct here — you want only prompt processing time, not generation. The server crashes on max_tokens=0.

Decode throughput (generation)

1. Start the server with the target parameter value
2. Send a prompt that generates a long response
3. Use max_tokens=16384 minimum (16k) — never max_tokens=1 for decode benchmarking. One token tells you nothing about sustained generation rate.
4. Measure predicted_per_second from the response timings object, or compute completion_tokens / wall_time

Why at least 16k tokens for decode

• Decode throughput varies over the generation (warmup, cache growth, MoE routing). A single token is noise.
• Speculative decoding acceptance rates stabilize over thousands of tokens.
• Short generations finish before the draft head reaches steady state.
• 16k tokens gives a stable average at ~22 tok/s (≈12 min per run).

Key Pattern

# Decode benchmark — measure generation throughput
body = json.dumps({
    "model": "<alias>",
    "messages": [{"role": "user", "content": "<prompt that generates 10k+ tokens>"}],
    "max_tokens": 16384,   # 16k minimum for decode; use max_tokens=1 ONLY for prefill
    "temperature": 0,     # deterministic for reproducibility
    "stream": False
})
# Measure wall time from request send to response receipt
# Get predicted_per_second from timings, or completion_tokens / wall_time

When to Use

• --threads-batch (prefill): use max_tokens=1
• --ubatch-size (decode): use max_tokens=16384+
• --spec-type / speculative decoding: use max_tokens=16384+ (decode only)
• Any server-side parameter that llama-bench cannot target

Benchmarking policy (MANDATORY)

1. STOP the production service before running any benchmark
2. Mask the service (systemctl mask, never disable — on these hosts disable deletes the unit file from /etc/systemd/system/)
3. Run the bench
4. Unmask + restart the service when done
5. Verify health before reporting results

Example Scripts

• scripts/bench-threads-batch-http.py — prefill sweep (max_tokens=1)
• scripts/bench-decode-threads-batch.py — decode sweep (max_tokens=256+)
• scripts/bench-spec-decode.py — speculative decoding config sweep