Frank
d104eef4e0
Every measurement behind the frank-medium runbook was taken with scripts that
lived in /tmp on two hosts — reproducible only until the next reboot, and
invisible to anyone reading the repository. This moves the useful four into
scripts/, where this repo already keeps its bench harnesses:
bench-vllm-corpus.py code-shaped corpus from the repo's own sources, because
synthetic random prompts misreport draft acceptance
badly (2.9-3.6 tok/update against ~5.8 on code)
bench-vllm-serve.sh concurrency sweep; encodes the two traps that produced
silent garbage — the tokenizer argument, without which
the client dies resolving "frank-medium" on HF and
reports no numbers at all, and the --ignore-eos
semantics that must not be compared across
bench-turn-latency.py per-turn TTFT for an agent-loop shape; the instrument
that decided the routing policy, and the one that shows
cold-vs-warm prefill directly
bench-llm-soak.py sustained multi-family load; what found the capacity
wall (three 30k-token families, 75% cold re-prefills,
zero failures)
Each header carries its traps and the interpretation; the runbook's Benchmarks
section now points at them instead of describing methodology with no tooling.
Also adds the missing half of the script seam: scripts/*.py are compiled, not
just scripts/*.sh syntax-checked — a harness with a syntax error could otherwise
ship unnoticed until someone needed it during an incident.
Both python harnesses were smoke-tested against the live fleet before commit:
the corpus builder wrote 15 prompts, and the turn harness measured 12.5 s cold
against 1.4 s warm on the same prompt — the cache behaviour the runbook claims.
45 lines
1.6 KiB
Bash
45 lines
1.6 KiB
Bash
#!/usr/bin/env bats
|
|
# Seam: script syntax.
|
|
# Every shell script in scripts/ must pass `bash -n`. When shellcheck is
|
|
# available it must pass with no errors (warnings tolerated in this repo).
|
|
|
|
load helpers
|
|
|
|
@test "all scripts pass bash -n" {
|
|
local any=0
|
|
while IFS= read -r s; do
|
|
any=1
|
|
bash -n "$s" || { echo "syntax error in $s"; return 1; }
|
|
done < <(find "$REPO_ROOT/scripts" -name '*.sh' -type f 2>/dev/null | sort)
|
|
[ "$any" -eq 1 ] || skip "no scripts yet"
|
|
}
|
|
|
|
@test "scripts are executable" {
|
|
while IFS= read -r s; do
|
|
[ -x "$s" ] || { echo "$s is not executable"; return 1; }
|
|
done < <(find "$REPO_ROOT/scripts" -name '*.sh' -type f 2>/dev/null | sort)
|
|
}
|
|
|
|
@test "shellcheck passes when available" {
|
|
if command -v shellcheck >/dev/null 2>&1; then
|
|
while IFS= read -r s; do
|
|
shellcheck -S error "$s" || { echo "shellcheck errors in $s"; return 1; }
|
|
done < <(find "$REPO_ROOT/scripts" -name '*.sh' -type f 2>/dev/null | sort)
|
|
else
|
|
skip "shellcheck not installed locally"
|
|
fi
|
|
}
|
|
|
|
@test "every python script in scripts/ compiles" {
|
|
# The shell half of this seam covered scripts/*.sh; the python half was only
|
|
# covered where a test happened to reference a specific file, so a syntax
|
|
# error in a committed harness could ship unnoticed until someone ran it
|
|
# during an incident.
|
|
local any=0
|
|
while IFS= read -r p; do
|
|
any=1
|
|
python3 -m py_compile "$p" || { echo "syntax error in $p"; return 1; }
|
|
done < <(find "$REPO_ROOT/scripts" -name '*.py' -type f 2>/dev/null | sort)
|
|
[ "$any" -eq 1 ] || skip "no python scripts yet"
|
|
}
|