---
name: llm-bench-noise-discipline
description: "Noise-floor discipline for single-run LLM endpoint A/B benchmarks: the ±3% law, replication-before-keeping, and bracket-confirmation of tuned values."
---

# LLM Bench Noise Discipline

Single-run A/B benchmarks against an LLM endpoint's own timings API have a much larger run-to-run noise floor than the metric's instantaneous precision suggests. On frank-primary (radiance, 256-token greedy decode at 43.6k-token prompt depth), three runs of the IDENTICAL config measured 33.06 / 32.03 / 30.90 t/s — a ~3% spread.

## The ±3% law
- Treat any single-run delta under ~3% as noise, regardless of the harness's reported confidence multiplier.
- Before believing a "keep", run the same config a second time (no edit, pure replication). If the replication drops below the noise band, flag the lucky draw as suspect (`flag_runs`) so best-metric math stops trusting it.
- Noise sources: TP all-reduce wire loss (wht6), page-cache warm/cold of the 51 GiB n-gram table, container boot ordering, expert-pool hit-rate warm-up.

## Bracket before trusting an optimum
A "tuned" value is only confirmed when BOTH neighboring steps are measured: one side may crash (boot wall) and the other may regress. Example at seqs=3: EXPERT_VS_CACHE 0.78 crash / 0.82 optimum / 0.84 = -9% (0.87 collapses to 0% per runbook); MNBT 4096 crash at seqs=3 while the seqs=2 A/B showed it bootable.

## Boot failures are data, not waste
A scratch arm that never reaches /health (720s timeout) maps the viable envelope. Three crashes at seqs=3 (0.78 weights-share, 4096-token chunks, bf16 wire) prove the production trio wht6/2048/0.82 is exactly what makes the 3-slot shape fit.

## Outside-noise signals found at this endpoint
- SPEC_TOKENS=4: -23% decode, draft acceptance 0.831->0.497 (the 4th draft token is rejected too often to pay its verification).
- SPEC_TOKENS=0: -6.8%; MTP drafting always pays.
- KV shape seqs=4/maxlen=100k (same 400k total): -11%; slot overhead is not paid by freed VRAM.

## Harness pattern that worked
stop prod -> boot scratch container on the SAME two GPUs with candidate argv -> poll /health -> ONE self-contained greedy request built from a fixed-seed repo corpus (deterministic prompt_n) -> parse timings -> tear down -> restore prod via trap (never mask a relinked unit; stop only). Fresh prefix-cache dir per run so the cold prefill is cold.