Frank
a34322aeb0
The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.
Surface check: the 25 on disk are exactly the skills the harness advertises
this session.
2.3 KiB
name, description
| name | description |
|---|---|
| llm-bench-noise-discipline | Noise-floor discipline for single-run LLM endpoint A/B benchmarks: the ±3% law, replication-before-keeping, and bracket-confirmation of tuned values. |
LLM Bench Noise Discipline
Single-run A/B benchmarks against an LLM endpoint's own timings API have a much larger run-to-run noise floor than the metric's instantaneous precision suggests. On frank-primary (radiance, 256-token greedy decode at 43.6k-token prompt depth), three runs of the IDENTICAL config measured 33.06 / 32.03 / 30.90 t/s — a ~3% spread.
The ±3% law
- Treat any single-run delta under ~3% as noise, regardless of the harness's reported confidence multiplier.
- Before believing a "keep", run the same config a second time (no edit, pure replication). If the replication drops below the noise band, flag the lucky draw as suspect (
flag_runs) so best-metric math stops trusting it. - Noise sources: TP all-reduce wire loss (wht6), page-cache warm/cold of the 51 GiB n-gram table, container boot ordering, expert-pool hit-rate warm-up.
Bracket before trusting an optimum
A "tuned" value is only confirmed when BOTH neighboring steps are measured: one side may crash (boot wall) and the other may regress. Example at seqs=3: EXPERT_VS_CACHE 0.78 crash / 0.82 optimum / 0.84 = -9% (0.87 collapses to 0% per runbook); MNBT 4096 crash at seqs=3 while the seqs=2 A/B showed it bootable.
Boot failures are data, not waste
A scratch arm that never reaches /health (720s timeout) maps the viable envelope. Three crashes at seqs=3 (0.78 weights-share, 4096-token chunks, bf16 wire) prove the production trio wht6/2048/0.82 is exactly what makes the 3-slot shape fit.
Outside-noise signals found at this endpoint
- SPEC_TOKENS=4: -23% decode, draft acceptance 0.831->0.497 (the 4th draft token is rejected too often to pay its verification).
- SPEC_TOKENS=0: -6.8%; MTP drafting always pays.
- KV shape seqs=4/maxlen=100k (same 400k total): -11%; slot overhead is not paid by freed VRAM.
Harness pattern that worked
stop prod -> boot scratch container on the SAME two GPUs with candidate argv -> poll /health -> ONE self-contained greedy request built from a fixed-seed repo corpus (deterministic prompt_n) -> parse timings -> tear down -> restore prod via trap (never mask a relinked unit; stop only). Fresh prefix-cache dir per run so the cold prefill is cold.