---
name: llm-serving-benchmark-methodology
description: "Measure and A/B a self-hosted multi-replica LLM endpoint honestly: corpus construction, per-arm salting, repeated samples, engine-counter semantics, and the traps that make results meaningless."
---

# Bench a multi-replica LLM deployment so the numbers mean something

Use when measuring or tuning a self-hosted LLM endpoint (vLLM, llama.cpp,
radiance/MXFP4), especially behind a proxy in front of several replicas.

## Set up a client that can actually send requests

`vllm bench serve` needs `--tokenizer <local model path>` with the model
directory mounted. Given only `--model <served-alias>` it tries to resolve that
alias on HuggingFace and dies before the first request — a run that looks like
it worked and produced nothing. Confirm with a single prompt before benching.

## Build the workload from real material

`--dataset-name random` generates token soup whose draft acceptance is nothing
like production (measured 2.9-3.6 tok/update against 5.8 on code), so it
mis-ranks speculative-decoding settings. Generate a JSONL corpus from the
repository's own sources — `{"prompt": "<instruction>\n\n<file body>"}` — and
fix output length with `--custom-output-len`. Keep two corpus shapes:

- **distinct prompts** (batch behaviour, prefill parallelism)
- **a shared long prefix with a varying tail** (conversation turns, where
  prefix caching decides everything)

## Salt every arm

Put a unique marker in the first bytes of every prompt, different per A/B arm.
Without it the second arm silently measures the first arm's warm cache and both
arms look identical.

## Measure the same workload before and after

Keep the workload fixed and change one knob. Repeat each arm — run-to-run
variance of 5-10 % is normal, and a single sample nearly made me ship the wrong
setting. Report medians per family/class, not one aggregate number.

## Read engine counters correctly

- vLLM's `Prefix cache hit rate:` line is a **lifetime average** across every
  prompt that replica ever saw. It cannot answer "is this conversation served
  warm?". Send the same prompt twice to an idle engine and compare TTFTs
  instead (measured: 22.3 s cold → 1.4 s warm for a 50k-token prefix).
- `GPU KV cache usage` tells you the pool is not full; it does not tell you a
  prefix survived. Capacity per card is the boot-time ledger line
  (`GPU KV cache size: N tokens`), not the card's VRAM.
- Reserve memory is not allocation: a bigger reserve does not free anything for
  anyone else, it only removes cache.

## Route-shape traps

A cold batch of distinct prompts prefills in parallel across replicas; the same
conversation's next turn only pays off if it lands where its prefix is cached.
These pull in opposite directions, so measure both shapes separately and route
by request shape (spread single-turn, pin continuations).

## When the answer is "no"

Write down rejected hypotheses with their numbers and the mechanism. Placement
balance, concurrency admission and chunk-size tweaks all looked obvious and all
measured flat or worse on this fleet; the wall turned out to be arithmetic
(cached prefixes plus live sequences share one pool). A runbook that records
that saves the next operator a night.