bench: commit the harnesses the frank-medium numbers came from
Every measurement behind the frank-medium runbook was taken with scripts that
lived in /tmp on two hosts — reproducible only until the next reboot, and
invisible to anyone reading the repository. This moves the useful four into
scripts/, where this repo already keeps its bench harnesses:
bench-vllm-corpus.py code-shaped corpus from the repo's own sources, because
synthetic random prompts misreport draft acceptance
badly (2.9-3.6 tok/update against ~5.8 on code)
bench-vllm-serve.sh concurrency sweep; encodes the two traps that produced
silent garbage — the tokenizer argument, without which
the client dies resolving "frank-medium" on HF and
reports no numbers at all, and the --ignore-eos
semantics that must not be compared across
bench-turn-latency.py per-turn TTFT for an agent-loop shape; the instrument
that decided the routing policy, and the one that shows
cold-vs-warm prefill directly
bench-llm-soak.py sustained multi-family load; what found the capacity
wall (three 30k-token families, 75% cold re-prefills,
zero failures)
Each header carries its traps and the interpretation; the runbook's Benchmarks
section now points at them instead of describing methodology with no tooling.
Also adds the missing half of the script seam: scripts/*.py are compiled, not
just scripts/*.sh syntax-checked — a harness with a syntax error could otherwise
ship unnoticed until someone needed it during an incident.
Both python harnesses were smoke-tested against the live fleet before commit:
the corpus builder wrote 15 prompts, and the turn harness measured 12.5 s cold
against 1.4 s warm on the same prompt — the cache behaviour the runbook claims.
This commit is contained in:
@@ -268,6 +268,15 @@ Restarting a stopped replica takes about 150 s (warm compile cache).
|
||||
|
||||
## Benchmarks
|
||||
|
||||
The harnesses live in the repository — the numbers below are reproducible with
|
||||
`scripts/bench-vllm-corpus.py` (build the code corpus), `scripts/bench-vllm-serve.sh`
|
||||
(concurrency sweep), `scripts/bench-turn-latency.py` (per-turn TTFT for an
|
||||
agent-loop shape, which is what the routing decision optimizes) and
|
||||
`scripts/bench-llm-soak.py` (sustained multi-family load). Each carries the
|
||||
traps in its header: the tokenizer argument that makes a `vllm bench` run report
|
||||
nothing, the salted prompts that keep a run from inheriting a warm cache, and
|
||||
why the corpus is code-shaped rather than `random`.
|
||||
|
||||
`vllm bench serve` against the proxy (both cards serving), random dataset,
|
||||
`--ignore-eos`, `2026-09-16`:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user