--- type: reference title: LLM Bench Windows and Measurement description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding. tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet] timestamp: 2026-10-05T00:00:00-07:00 --- # LLM Bench Windows and Measurement Fleet law first: **STOP the production service before any GPU experiment** (AGENTS.md Benchmarking Policy) — a bench beside a live service starves the GPU and corrupts results; an unisolated GPU experiment has precedent for wedging a host until power cycle. Reference harnesses: `scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`, `scripts/autoresearch/frank-archon-arms.sh`. ## 1. The window structure (one config per arm, a driver owns the window) 1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts EXITED containers and can never clear — `rm -f` the exited one first) AND port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01: `ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s then polls `/health` gets **200 from the still-running PRODUCTION engine** — the arm "passes" with production's boot log, then `podman exec` dies with `container state improper`. 2. **Boot the arm as the service, not a scratch port** (no scratch ports on LLM hosts — one card, one server). The serve script takes its config via env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`) pointing at a staged copy; the checkout is never mutated. 3. **Fast-fail the health wait**: poll `podman ps | grep -qx ` inside the loop — a refused boot dies in one poll instead of 20 min. Capture the serve script's FULL stdout on failure (a `tail -8` loses engine-init root causes; raise happens above the tail). 4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots, SpecDecoding lines) — they survive measure crashes. 5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills), `podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver before the next boot; restore production from the EXIT trap and verify the **conjunction oracle**: health 200 + `is-active` + `is-enabled` + mask-symlink clean. A bare `systemctl start` is not a restore oracle. (`systemctl mask` FAILS on a relinked unit — log it, explicit stop is enough: systemd never auto-restarts an explicitly stopped unit.) 6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for mixed speeds. **Destructive tails (2026-10-01):** a scratch script whose tail does `systemctl stop` + `podman rm -f ` fires later if a probe loop survived the driver kill — it can destroy a just-deployed production engine (signature: deploy exits green, then seconds-to-minutes later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not crashed). Before ANY deploy: `pgrep -af` every scratch script of the session and kill strays by explicit PID; killing the dev-side job does NOT kill the host-side `deploy-remote.sh` (two of them once managed the same unit). Two failure shapes inside the window that read as "the bench died": - **Stale reads from a stopping container.** `podman stop` answers while the engine inside is still flushing; a `podman logs` taken in that window returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or `podman wait`) before reading logs, or you will write "no results" into the table for a run that has them. - **A D-state worker wedges the stop.** A bench child that wedged in the kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and then the container is `Stopping` forever. Kill by container first (`podman kill ` → SIGKILL path), verify the process is gone from `podman ps` AND `nvidia-smi`, and only then re-serve. Never let the window script proceed to a redeploy with a zombie bench worker holding VRAM. ## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth) **What llama-bench cannot express — decide BEFORE designing:** no `--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove activation buffers fit beside the DEPLOYED KV — only the real unit answers that), no `--parallel` (no slot shape, no concurrency), no `--spec-*` (`tg` numbers are the drafterless baseline, useful as a control only), and a warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at ~zero depth and meaningless for long conversations. **The separator trap:** `/` separates devices WITHIN one configuration, `,` separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the single value 36 → everything on one card. `--tensor-split 36,29` (server) → `-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`. **Harness shape:** one invocation per cell (an infeasible cell calls `exit(1)` and kills every later cell of a cross-product invocation — and infeasibility is usually the question); TSV row per cell as it completes + `bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi --showmeminfo vram` VRAM trace per cell (a cell that allocated less than intended looks fast); resume by skipping non-empty cell outputs; stop the service in the harness, restore from `trap … EXIT`, verify afterwards. Match the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`). **Run under a transient unit, not a shell:** ```bash sudo systemd-run --unit= --collect --no-block --property=Type=oneshot \ --property=LimitMEMLOCK=infinity --property=User=dank \ /home/dank/serverAppConfigs/scripts/.sh ``` Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has 8 MiB and fails the load for an unrelated reason. This also survives ssh disconnects. **Boot rungs under the production environment, not a bench shell** — a rootless shell probe booted 3×262144 bf16 that production refused (systemd `User=` + memlock change what the pinned-expert path holds): `sudo systemd-run --uid= --unit= -p LimitMEMLOCK=infinity -p Restart=no env VARFILE=… /path/serve.sh`. **Verify before burning hours:** dry-run against a STUB binary with `SERVICE=` (the trap/stop never reached), tiny grid, including one forced-`exit 3` cell — prove the TSV row, failure capture, resume logic. Validate the real flag set in ~1 s by pointing `-m` at a nonexistent file. **Reading:** bracket the sweep (reference cell first AND last); check per-cell `stddev_ts` (a difference smaller than spread is not a result); single-sequence decode is usually flat across batch/ubatch (prefill knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating smaller-batch cells); report what the grid cannot prove (drafter, slots, real context). ## 3. HTTP benches of server-side parameters `llama-bench` benches the bench binary, not the running server — server-side params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need HTTP: - **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure `prompt_tokens / wall_time`. - **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and spec acceptance all stabilize over thousands of tokens); measure `predicted_per_second` from `timings` or `completion_tokens / wall_time`; temperature 0. Known fleet harnesses: `scripts/bench-threads-batch-http.py`, `scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`. sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3 reasoning parser streams zero deltas — use `/v1/completions`. ## 4. The measurement law (under speculative decoding this is absolute) **Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token mapping; chunk counting reported 8.3 tok/s for a config whose server counters said 23–26 — a 3x false regression that nearly abandoned the change that worked. Always: ```python decode_tok_s = usage.completion_tokens / (wall - ttft) # stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk ``` vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a measurement (6.2 → 13.1 across adjacent windows on identical settings) — use them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean acceptance length`, per-position rates). Warm pass + ≥3 runs, median; temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions (ct < 256) and acceptance > 90 % is degenerate generation** (the fork's repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample nearly shipped the wrong setting. ## 5. Honest methodology - **Match the corpus to the workload.** Token soup under-reports speculative decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft settings; acceptance, not step time, moves throughput. Build the corpus from the repo's own sources: `{"prompt": "\n\n"}`, `--custom-output-len`. - **Salt every run** with a unique per-arm token in the first bytes — it is the prefix-cache key (and a proxy routing key). Without it arm 2 measures arm 1's warm cache. - **Two workload shapes need two clients:** distinct prompts (batch behaviour, prefill parallelism) and sequential turns with growing history (TTFT of the first streamed chunk per turn — the only measurement that exposes prefix-cache behaviour). These pull in opposite directions: a cold batch spreads prefill across replicas, a conversation's next turn only pays if it lands where its prefix is cached. - **Read the engine's ledger, not the card:** `Available KV cache memory` / `GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line is a LIFETIME average across every prompt the replica ever saw — send the same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold → 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not full; it does NOT say a prefix survived. The KV budget is NOT monotonic in chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure three points before treating a knob as a lever. - **Soak before believing:** 15–30 min of concurrent families, count failures AND latency; report hit/miss split and medians with min/max — the median of a bimodal distribution describes neither mode. - **Verification that is not the benchmark:** run the client the fleet actually uses; and a deployed unit is not proven until an unattended reboot brings it back. - **Record the "no" answers** with numbers and mechanism — placement balance, concurrency admission and chunk-size tweaks all measured flat or worse here; the wall was arithmetic (cached prefixes + live sequences share one pool). ## 6. Sweep hygiene (the arms that produced wrong verdicts) - **Prefetch uniformity:** every arm in a sweep carries the same expert-prefetch setting. The 2026-09-29 campaign measured losing arms with prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec 12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed verdict. - **Knob-landed preflight:** for every `-e` knob, confirm `podman exec printenv ` — a knob the fork silently ignores produces a "parity" arm that is really the default (v1 of the gather sweep lost all 3 arms this way; serve-script hooks are additive and can drop vars). - **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers compare against default-off, not "incumbent + one flag". - **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than autofit (remainder) so pool numbers are reproducible; compute need = tokens × measured B/token (hybrid models: KV dtype does NOT double mamba states — measure, don't assume 2x). - **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly deployed a 2x-prefill-tax config. Per-depth tables, never a single median — one fork's gather win grows with depth (54 t/s @1k, 39.5 @short). - Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy `mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them. ## 7. Bash laws that ate windows (each cost an arm) 1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word → `SPEC=3: command not found`, nothing runs. Pass literal assignments: `SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty. 2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside an if/else it aborts ONLY that compound statement — the script CONTINUES, every arm tears down clean with zero measurements. Detection law: `ls` the per-arm `.measure.txt` files before trusting any summary; arm echoes are not evidence. 3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u` it aborts AFTER the harness stopped production. Two statements. 4. `sed -i` renames the file; already-running bash keeps the OLD inode — mid-flight patches miss arms launched in the preceding minutes. Patch early or run a make-up window. 5. Never heredoc python through ssh→zsh→bash — write to a file, ship it (`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting fails silently. Same trap in `git commit -m`: backticks become command substitution. 6. No comments INSIDE a backslash-continued command (the continuation joins the comment); verify multi-line command construction with `bash -n` AND a render test. ## 8. Results location and recovery (frank-archon arms) | copy | path | |---|---| | per-arm evidence | `$ARMS_DIR` = `~/.jobs//` (`.rc`, `.done` beside it) | | summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `.out/.env/.overlay/.rc/.metrics` | | server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` | | durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) | | fallback | `~/.omp/agent/sessions///.{shake,bash}.log` (permanent) | `/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests — constants 99.9/88.8 identical across arms by construction; **identical numbers across arms == stub or bug, never a measurement.** After compaction, re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per window. ## Related - [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept - [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes - [[LLM Performance Diagnosis]] — profiling and attribution methods - [[Remote Job Runner]] — detached execution of long sweeps - [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history