Frank
a34322aeb0
The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.
Surface check: the 25 on disk are exactly the skills the harness advertises
this session.
23 lines
1.7 KiB
Markdown
23 lines
1.7 KiB
Markdown
---
|
|
name: frank-primary-decode-tuning
|
|
description: "Tune frank-primary (radiance TP=2) decode throughput: benchmark harness usage, flag levers, known optima and boot-failure walls."
|
|
---
|
|
|
|
# frank-primary decode tuning
|
|
|
|
Harness: `bash autoresearch.sh` (dev box) scp's `scripts/frank-primary.env` + `scripts/autoresearch-frank-primary.sh` to dankserver, stops prod (`systemctl stop frank-primary`, NEVER mask — relinked unit), boots scratch container `arbench` on :7770 (fresh prefix-cache tier at `$STATE_DIR/kvcache/arbench` for cold prefill), polls health 720s, sends ONE greedy completion (fixed 140k-char repo-corpus prompt -> 43,627 tokens, 256-token decode), parses `timings` into METRIC lines, tears down, restores prod via EXIT trap. ~4 min/run; decode noise ~0.3%.
|
|
|
|
Metric: `METRIC decode_tps` = `timings.predicted_per_second` at 43.6k depth.
|
|
|
|
## Optima (measured 2026-10-06, keep these)
|
|
- `FRANK_PRIMARY_SPEC_TOKENS=2` (unimodal: 1->31.89, 2->32.40, 3->31.72, 4->24.57 t/s; acceptance collapses past 3)
|
|
- `FRANK_PRIMARY_MAX_SEQS=3` with `MAX_MODEL_LEN=200000` (600k KV; +4.2% total; 4 slots regresses)
|
|
- `FRANK_PRIMARY_EXPERT_VS_CACHE=0.82` bracket-confirmed (0.78 fails boot, 0.84 -8.9%, 0.87 collapse)
|
|
- `TP_WIRE=wht6`, `MAX_BATCHED_TOKENS=2048` — REQUIRED at seqs=3: bf16 wire and MNBT 4096 both fail boot (VRAM)
|
|
|
|
## Non-levers (decode-neutral, don't spend runs)
|
|
GPU_HEADROOM_MIB (48 = neutral), HOST_POOL_MIB 49152 (neutral), MAX_MODEL_LEN any change (regresses: bigger KV steals resident weights), prefix-cache tiers (prefill-side).
|
|
|
|
## Rules
|
|
One flag per iteration; keep only > +1% decode over best; a boot failure is `crash` (trap restores prod). Constraint floor: MAX_SEQS*MAX_MODEL_LEN >= 400000.
|