Frank
a34322aeb0
The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.
Surface check: the 25 on disk are exactly the skills the harness advertises
this session.
3.1 KiB
3.1 KiB
name, description
| name | description |
|---|---|
| radiance-frank-primary-benching | Bench frank-primary (radiance) decode/prefill via the scratch-container harness: method, noise floor, and the MNBT/HOST_POOL/envelope laws to respect before proposing any flag change. |
Benching frank-primary (radiance)
Method (works as of 2026-10-06)
- Entry point:
bash autoresearch.sh(dev box) → scp'sscripts/frank-primary.env+scripts/autoresearch-frank-primary.shto dankserver → the bench script ON dankserver: stops frank-primary (relinked unit:stop, nevermask), boots scratch containerarbenchon the candidate env, polls /health (binds after full load, ~110 s cold), runs ONE greedy completion (fixed ~140k-char repo-corpus prompt, seed 1234 → 43,627 prompt tokens, 256 decode tokens, temp 0), parses the enginetimingsobject, tears down, and restores prod via a trap. - Metrics from
timings:prompt_per_second(cold prefill),predicted_per_second(decode at depth = (predicted_n-1)/predicted_ms),prompt_ms(TTFT),draft_n/draft_n_accepted(acceptance). - Fresh prefix-cache tier per run (
/data/kvcache/arbench) so cold prefill is cold; prod's tier untouched. - Bench locally on dankserver (127.0.0.1) — no LAN jitter. Run-to-run ~4-5 min.
Noise floor (measurement law)
- Decode: ~±3% on IDENTICAL config (33.06 / 32.03 / 30.90 observed). Single-run deltas <3% are not evidence; re-run before keeping.
- Prefill: ~±2% (453/448/457 identical config).
Laws at the seqs=3 / maxlen=200k / fp8-KV / MTP-on shape (2×R9700, TP=2)
- MNBT is a sharp resonance at 2048: 2560 → decode 2.6-7.7 t/s (acceptance stays normal, verify stalls ~130 ms/step, generation truncates); 1536 → decode 24.7 + 22% prefill loss + truncation; 4096 → boot failure at seqs=3 (boots at seqs=2, -2.6% only). DO NOT move MNBT off 2048.
- HOST_POOL_MIB 40960 is the load-bearing floor: 36864 fails boot; 49152 boots but decode-flat.
- EXPERT_VS_CACHE 0.82 is bracket-confirmed: 0.78 fails boot; 0.84 costs 8.9% (toward the documented 0.87 collapse cliff).
- TP_WIRE wht6 is mandatory at seqs=3: bf16 (lossless) fails boot.
- SPEC_TOKENS: depths 1-3 statistically tied on decode; 0 = -6.8% (real), 4 = -22.5% (acceptance collapses to 0.50). Depth 2 nominal peak.
- MAX_MODEL_LEN 262144 (native) costs 2.5% — the larger KV pool steals resident-weights VRAM at expert_tiered.
- GPU_HEADROOM 48 vs 96: no effect.
Hazard: abandoned bench contamination
- If a prior bench run is skipped/abandoned mid-flight, the next scratch boot may race the leftover engine and produce garbage numbers. Before trusting any collapse measurement:
podman ps -aon dankserver — confirm the previous arbench is gone and frank-primary is active — then reproduce the collapse on a clean boot before writing it into the record.
State
- HEAD on the autoresearch branch: spec=2, seqs=3, maxlen=200k (df6b5bb6e807). All spec 1-3 × seqs 2-3 configs decode within 30.9-33.1 — statistically tied. Frank's decision pending: keep spec=2/seqs=3 or revert to his spec=3/seqs=2 (83% acceptance, likely better for real multi-turn traffic).
- Production verified healthy (health 200) after every run via the harness trap.