---
name: live-llm-benchmark-hygiene
description: "Measure and tune a live LLM server without fooling yourself: workload-matched corpora, salted runs against warm prefix caches, turn-level clients, engine ledger for capacity, soak tests, and reboot verification."
---

# Benchmarking a live LLM server honestly

Use when measuring or tuning a served model (vLLM, llama.cpp, halogen, radiance,
any OpenAI-compatible endpoint). The point is to avoid the four confounds that
turned a night's numbers into fiction before each one was fixed.

## 1. Match the corpus to the workload

Synthetic token soup (`vllm bench serve --dataset-name random`) under-reports
speculative decoding badly: measured 2.9-3.6 tok/update against 3.86 on real
code, and acceptance — not step time — is what moves throughput. For a coding
fleet, build the corpus from the repo's own sources: file body plus an
instruction that requires reading it. Then `tok/update` and TPOT mean something.

## 2. Salt every run so the prefix cache cannot help

vLLM runs with `--enable-prefix-caching`, and a repeated corpus silently
measures a warm cache. Put a unique per-run token in the first bytes (it is both
the engine's cache key and, if a proxy routes on it, the routing key). Without
this, the second run of any comparison is not comparable to the first.

## 3. Measure turns, not only batches

A batch of distinct prompts parallelises prefill and hides the real cost of a
conversation: an agent loop re-sends its context every turn. Two workloads need
two clients:

- **batch/throughput** — `vllm bench serve --dataset-name custom --dataset-path
  <jsonl>` with `--custom-output-len`, several concurrency levels.
- **sequential turns** — a small client that sends turn after turn with growing
  history and a large shared prefix, timing **TTFT of the first streamed chunk**
  per turn. This is the only measurement that exposes prefix-cache behaviour.

## 4. Read the engine's own ledger for capacity

Do not infer memory from the card's size. The server prints the truth at boot:

```
Available KV cache memory: 5.44 GiB
GPU KV cache size: 122,636 tokens
Maximum concurrency for 114,688 tokens per request: 1.07x
```

That is the number to quote for "how many concurrent conversations fit".
Related trap: the arena/KV budget is **not monotonic in chunk size** — on one
host, available KV was 5.44 GiB at `CHUNK=8192`, 4.22 GiB at 16384 and 4.16 GiB
at 4096 (where it would not even boot). Never assume "smaller chunk frees KV";
measure three points before treating a knob as a lever.

## 5. Soak before believing a tuning change

Run several families concurrently for 15-30 minutes and count failures, not just
latency: a nightly soak returned 155/155 answers with zero failovers while only
25% of turns hit the cache — which is how the capacity ceiling was found as a
*latency* effect rather than an error. Report hit/miss split, medians, and the
min/max, because the median of a bimodal distribution describes neither mode.

## 6. Verification that is not the benchmark

- Run the client the fleet actually uses (e.g. `omp --model <alias> -p
  --no-tools`) — a config file that parses is not a config that works, and a
  non-obvious API mode (raw `/v1/completions`) may still be fine because the
  client templates its own prompt.
- A deployed inference unit is not proven until an **unattended reboot** brings
  it back: verify `is-active`, `/health`, and one real completion afterwards.