Frank
a34322aeb0
The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.
Surface check: the 25 on disk are exactly the skills the harness advertises
this session.
33 lines
3.1 KiB
Markdown
33 lines
3.1 KiB
Markdown
---
|
||
name: radiance-frank-primary-benching
|
||
description: "Bench frank-primary (radiance) decode/prefill via the scratch-container harness: method, noise floor, and the MNBT/HOST_POOL/envelope laws to respect before proposing any flag change."
|
||
---
|
||
|
||
# Benching frank-primary (radiance)
|
||
|
||
## Method (works as of 2026-10-06)
|
||
- Entry point: `bash autoresearch.sh` (dev box) → scp's `scripts/frank-primary.env` + `scripts/autoresearch-frank-primary.sh` to dankserver → the bench script ON dankserver: stops frank-primary (relinked unit: `stop`, never `mask`), boots scratch container `arbench` on the candidate env, polls /health (binds after full load, ~110 s cold), runs ONE greedy completion (fixed ~140k-char repo-corpus prompt, seed 1234 → 43,627 prompt tokens, 256 decode tokens, temp 0), parses the engine `timings` object, tears down, and restores prod via a trap.
|
||
- Metrics from `timings`: `prompt_per_second` (cold prefill), `predicted_per_second` (decode at depth = (predicted_n-1)/predicted_ms), `prompt_ms` (TTFT), `draft_n`/`draft_n_accepted` (acceptance).
|
||
- Fresh prefix-cache tier per run (`/data/kvcache/arbench`) so cold prefill is cold; prod's tier untouched.
|
||
- Bench locally on dankserver (127.0.0.1) — no LAN jitter. Run-to-run ~4-5 min.
|
||
|
||
## Noise floor (measurement law)
|
||
- Decode: ~±3% on IDENTICAL config (33.06 / 32.03 / 30.90 observed). Single-run deltas <3% are not evidence; re-run before keeping.
|
||
- Prefill: ~±2% (453/448/457 identical config).
|
||
|
||
## Laws at the seqs=3 / maxlen=200k / fp8-KV / MTP-on shape (2×R9700, TP=2)
|
||
- **MNBT is a sharp resonance at 2048**: 2560 → decode 2.6-7.7 t/s (acceptance stays normal, verify stalls ~130 ms/step, generation truncates); 1536 → decode 24.7 + 22% prefill loss + truncation; 4096 → boot failure at seqs=3 (boots at seqs=2, -2.6% only). DO NOT move MNBT off 2048.
|
||
- **HOST_POOL_MIB 40960 is the load-bearing floor**: 36864 fails boot; 49152 boots but decode-flat.
|
||
- **EXPERT_VS_CACHE 0.82 is bracket-confirmed**: 0.78 fails boot; 0.84 costs 8.9% (toward the documented 0.87 collapse cliff).
|
||
- **TP_WIRE wht6 is mandatory** at seqs=3: bf16 (lossless) fails boot.
|
||
- **SPEC_TOKENS**: depths 1-3 statistically tied on decode; 0 = -6.8% (real), 4 = -22.5% (acceptance collapses to 0.50). Depth 2 nominal peak.
|
||
- MAX_MODEL_LEN 262144 (native) costs 2.5% — the larger KV pool steals resident-weights VRAM at expert_tiered.
|
||
- GPU_HEADROOM 48 vs 96: no effect.
|
||
|
||
## Hazard: abandoned bench contamination
|
||
- If a prior bench run is skipped/abandoned mid-flight, the next scratch boot may race the leftover engine and produce garbage numbers. Before trusting any collapse measurement: `podman ps -a` on dankserver — confirm the previous arbench is gone and frank-primary is active — then reproduce the collapse on a clean boot before writing it into the record.
|
||
|
||
## State
|
||
- HEAD on the autoresearch branch: spec=2, seqs=3, maxlen=200k (df6b5bb6e807). All spec 1-3 × seqs 2-3 configs decode within 30.9-33.1 — statistically tied. Frank's decision pending: keep spec=2/seqs=3 or revert to his spec=3/seqs=2 (83% acceptance, likely better for real multi-turn traffic).
|
||
- Production verified healthy (health 200) after every run via the harness trap.
|