bench: commit the harnesses the frank-medium numbers came from
Every measurement behind the frank-medium runbook was taken with scripts that
lived in /tmp on two hosts — reproducible only until the next reboot, and
invisible to anyone reading the repository. This moves the useful four into
scripts/, where this repo already keeps its bench harnesses:
bench-vllm-corpus.py code-shaped corpus from the repo's own sources, because
synthetic random prompts misreport draft acceptance
badly (2.9-3.6 tok/update against ~5.8 on code)
bench-vllm-serve.sh concurrency sweep; encodes the two traps that produced
silent garbage — the tokenizer argument, without which
the client dies resolving "frank-medium" on HF and
reports no numbers at all, and the --ignore-eos
semantics that must not be compared across
bench-turn-latency.py per-turn TTFT for an agent-loop shape; the instrument
that decided the routing policy, and the one that shows
cold-vs-warm prefill directly
bench-llm-soak.py sustained multi-family load; what found the capacity
wall (three 30k-token families, 75% cold re-prefills,
zero failures)
Each header carries its traps and the interpretation; the runbook's Benchmarks
section now points at them instead of describing methodology with no tooling.
Also adds the missing half of the script seam: scripts/*.py are compiled, not
just scripts/*.sh syntax-checked — a harness with a syntax error could otherwise
ship unnoticed until someone needed it during an incident.
Both python harnesses were smoke-tested against the live fleet before commit:
the corpus builder wrote 15 prompts, and the turn harness measured 12.5 s cold
against 1.4 s warm on the same prompt — the cache behaviour the runbook claims.
This commit is contained in:
@@ -29,3 +29,16 @@ load helpers
|
||||
skip "shellcheck not installed locally"
|
||||
fi
|
||||
}
|
||||
|
||||
@test "every python script in scripts/ compiles" {
|
||||
# The shell half of this seam covered scripts/*.sh; the python half was only
|
||||
# covered where a test happened to reference a specific file, so a syntax
|
||||
# error in a committed harness could ship unnoticed until someone ran it
|
||||
# during an incident.
|
||||
local any=0
|
||||
while IFS= read -r p; do
|
||||
any=1
|
||||
python3 -m py_compile "$p" || { echo "syntax error in $p"; return 1; }
|
||||
done < <(find "$REPO_ROOT/scripts" -name '*.py' -type f 2>/dev/null | sort)
|
||||
[ "$any" -eq 1 ] || skip "no python scripts yet"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user