command-center/agent/managed-skills/autoresearch-llm-bench-window/SKILL.md

name, description

name	description
autoresearch-llm-bench-window	How to run an iterative config-tuning window on a Dank-fleet LLM endpoint: harness pattern, noise-floor-first discipline, bracket mapping, and the frank-primary laws measured in the 2026-10-06 segment.

Autoresearch bench window for a fleet LLM endpoint

Use when running (or re-running) an iterative config-tuning segment against a Dank-fleet LLM endpoint (frank-primary/frank-large/frank-small): build the harness, establish the noise floor, then bracket-map one flag at a time.

Harness pattern (already built for frank-primary — reuse, don't rebuild)

Committed at autoresearch.sh (dev box) + scripts/autoresearch-frank-primary.sh (dankserver) in serverAppConfigs. The pattern generalizes to any fleet endpoint:

1. Dev-box wrapper scp's the candidate env file + bench script to the target host, then ssh-runs the bench. Never push the candidate config to origin (the scp dirties the host checkout on purpose).
2. Host script, in order: stop the production unit (relinked units: systemctl stop, NEVER mask — mask refuses a symlinked unit file) → boot a scratch container (distinct name, e.g. arbench) on the SAME devices with the candidate env — mirror the production podman contract exactly, differing only in container name and a FRESH prefix-cache tier (wiped per run) so cold measurements stay cold → poll the health endpoint until it answers (the listener binds AFTER the full load, so a bare curl is honest; allow ~2-3 min cold boot, 720s timeout) → run ONE self-contained deterministic request → parse the engine's timings object into METRIC lines → tear down the scratch → restore production.
3. The restore MUST be an EXIT trap (restore() { podman rm -f scratch; systemctl start <unit>; }) so a failed bench never leaves the endpoint down. Verify after EVERY run: systemctl is-active, health curl until it returns JSON (200 right after start is NOT enough — it can bind 60-90s later), and no scratch leftover.
4. Bench request: fixed prompt built from a sorted repo-corpus glob with a fixed random seed (deterministic → stable prompt_n), greedy sampling (temperature 0) for a fixed decode length, sent locally (127.0.0.1) to remove LAN jitter. Read prefill/decode/draft-acceptance from the engine's own timings object — never roll a custom bench without asking.
5. Scratch containers must die by teardown; never leave a script whose tail stops/removes a production unit (the stale-probe-tail lesson).

Discipline (the laws that make the numbers mean anything)

1. Noise floor FIRST. Before iterating, replicate an identical config 2-3x and treat that spread as the floor. Fleet endpoints here: decode ±3%, prefill ±2%. A single-run "win" under the floor is NOT a win — this exact mistake cost a false 33.06 keep in the 2026-10-06 segment.
2. One flag change per iteration; log every run honestly (keep/discard/ crash); discards auto-revert the worktree.
3. Bracket-map, don't climb. Test both directions of every lever around its deployed value, INCLUDING boot failures — crashes are envelope data ("0.82/2048/wht6/40960 is the seqs=3-viable quartet" came from four crash walls). Test downward too: 40960 MiB host pool is a floor, not slack.
4. Repro before recording a mechanism. An anomalous result next to an abandoned bench may be contamination (boot race). Re-run clean once; only then name a law. The MNBT 2560 collapse (2.6 t/s) was reproduced clean to rule this out.
5. Watch for truncation. predicted_n < max_tokens means the generation died mid-run — that config is pathological regardless of the rate.
6. When the sweep ties, tie-break on real-traffic properties (e.g. draft acceptance for multi-turn workloads), never on probe artifacts, and prefer the config that matches the documented runbook.

frank-primary specifics (2026-10-06 segment, commit c80c250)

• MNBT=2048 is a RESONANCE POINT at MTP-on shapes, not an optimum: 2560 collapses decode to 2.6-7.7 t/s (acceptance normal, verify stalls ~130ms/step, truncated output), 1536 truncates at 57/256, 4096 won't boot at seqs=3. Do not re-test.
• HOST_POOL_MIB 40960 is the load-bearing floor (36864 crashes boot).
• Spec depth 1-3 and seqs 2-3 all tie within noise; 0 and 4 are real regressions (29.57 / 24.57 t/s vs 31.72 baseline).
• Levers known flat: GPU_HEADROOM, HOST_POOL upward, maxlen above 200k.
• Final kept config = Frank's original (spec=3, seqs=2, maxlen=200k, wht6, 0.82, 2048, 40960, fp8 KV, TP=2), baseline decode 31.72 t/s at 43.6k depth.