Four harness gaps that block an honest MTP re-measurement:
- measure: AR_TEMPS sweeps sampling temperatures (default "0,0.7"). Draft
ACCEPTANCE is sampling-dependent — a greedy draft matches a greedy target
exactly and is rejected more under a non-greedy target — so judging a spec
arm only at temperature 0 judges half the contract. The unsuffixed keys stay
temperature 0, so every historical number remains comparable.
- window: capture an evidence.log sidecar (expert-plan lines, compile path,
graph sizes, /metrics spec-decode and cache counters, OOM signatures). The
arena/overhead/reserve numbers exist ONLY in the boot log; an arm that loses
decode speed is otherwise unexplainable afterwards.
- orchestrator: FRANK_PRIMARY_AR_ENV_FILE stages an env other than the repo's,
which is how arms A/B without mutating the checkout (the fingerprint covers
whatever is staged, so the memo can never answer one arm with another's
numbers), plus METRIC lines for the swept temperatures and the evidence dump.
- scripts/autoresearch/frank-primary-arms.sh: runs a sequence of full windows
(baseline, MTP3+probabilistic with a step-4 capture ladder at precise memory
ON, the same at the default overhead, and the vendor SEQS-16 shape), gating
each staged env on its JSON knobs still parsing AFTER sourcing — an unquoted
{"temperature":0.7} loses its quotes at source time and reaches the engine as
{temperature:0.7}, which is a 20-minute boot wasted.
Benchmark entrypoint: bash autoresearch.sh
Goal: Make frank-primary faster: raise decode tok/s on the tcclaviger vLLM endpoint (dankserver :7770, Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700, automatic tiered-expert offload). Context size may be altered (MAXLEN/SEQS/KV_BYTES and friends); quantization may NOT be altered (the 131-shard FP8 payload and the digest-pinned image are frozen).
Frank