serverAppConfigs/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md

---
type: reference
title: LLM Bench Windows and Measurement
description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
timestamp: 2026-10-05T00:00:00-07:00
---

# LLM Bench Windows and Measurement

Fleet law first: **STOP the production service before any GPU experiment**
(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
GPU and corrupts results; an unisolated GPU experiment has precedent for
wedging a host until power cycle. Reference harnesses:
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
`scripts/autoresearch/frank-primary-arms.sh`.

## 1. The window structure (one config per arm, a driver owns the window)

1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
   no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
   EXITED containers and can never clear — `rm -f` the exited one first) AND
   port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
   `ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
   then polls `/health` gets **200 from the still-running PRODUCTION engine**
   — the arm "passes" with production's boot log, then `podman exec` dies
   with `container state improper`.
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
   LLM hosts — one card, one server). The serve script takes its config via
   env-file override (`FRANK_SMALL_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
   pointing at a staged copy; the checkout is never mutated.
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
   the loop — a refused boot dies in one poll instead of 20 min. Capture the
   serve script's FULL stdout on failure (a `tail -8` loses engine-init root
   causes; raise happens above the tail).
4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
   SpecDecoding lines) — they survive measure crashes.
5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
   `podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
   before the next boot; restore production from the EXIT trap and verify the
   **conjunction oracle**: health 200 + `is-active` + `is-enabled` +
   mask-symlink clean. A bare `systemctl start` is not a restore oracle.
   (`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
   enough: systemd never auto-restarts an explicitly stopped unit.)
6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
   mixed speeds.

**Destructive tails (2026-10-01):** a scratch script whose tail does
`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
probe loop survived the driver kill — it can destroy a just-deployed
production engine (signature: deploy exits green, then seconds-to-minutes
later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
and kill strays by explicit PID; killing the dev-side job does NOT kill the
host-side `deploy-remote.sh` (two of them once managed the same unit).

Two failure shapes inside the window that read as "the bench died":

- **Stale reads from a stopping container.** `podman stop` answers while the
  engine inside is still flushing; a `podman logs` taken in that window
  returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
  `podman wait`) before reading logs, or you will write "no results" into
  the table for a run that has them.
- **A D-state worker wedges the stop.** A bench child that wedged in the
  kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
  then the container is `Stopping` forever. Kill by container first
  (`podman kill <ctr>` → SIGKILL path), verify the process is gone from
  `podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
  window script proceed to a redeploy with a zombie bench worker holding
  VRAM.

## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)

**What llama-bench cannot express — decide BEFORE designing:** no
`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
activation buffers fit beside the DEPLOYED KV — only the real unit answers
that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
(`tg` numbers are the drafterless baseline, useful as a control only), and a
warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
~zero depth and meaningless for long conversations.

**The separator trap:** `/` separates devices WITHIN one configuration, `,`
separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
single value 36 → everything on one card. `--tensor-split 36,29` (server) →
`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.

**Harness shape:** one invocation per cell (an infeasible cell calls
`exit(1)` and kills every later cell of a cross-product invocation — and
infeasibility is usually the question); TSV row per cell as it completes +
`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
intended looks fast); resume by skipping non-empty cell outputs; stop the
service in the harness, restore from `trap … EXIT`, verify afterwards. Match
the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).

**Run under a transient unit, not a shell:**

```bash
sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
     --property=LimitMEMLOCK=infinity --property=User=dank \
     /home/dank/serverAppConfigs/scripts/<harness>.sh
```

Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
8 MiB and fails the load for an unrelated reason. This also survives ssh
disconnects. **Boot rungs under the production environment, not a bench
shell** — a rootless shell probe booted 3×262144 bf16 that production refused
(systemd `User=` + memlock change what the pinned-expert path holds):
`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
Restart=no env VARFILE=… /path/serve.sh`.

**Verify before burning hours:** dry-run against a STUB binary with
`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
including one forced-`exit 3` cell — prove the TSV row, failure capture,
resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
nonexistent file. **Reading:** bracket the sweep (reference cell first AND
last); check per-cell `stddev_ts` (a difference smaller than spread is not a
result); single-sequence decode is usually flat across batch/ubatch (prefill
knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
smaller-batch cells); report what the grid cannot prove (drafter, slots, real
context).

## 3. HTTP benches of server-side parameters

`llama-bench` benches the bench binary, not the running server — server-side
params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
HTTP:

- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
  `prompt_tokens / wall_time`.
- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
  spec acceptance all stabilize over thousands of tokens); measure
  `predicted_per_second` from `timings` or `completion_tokens / wall_time`;
  temperature 0.

Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
reasoning parser streams zero deltas — use `/v1/completions`.

## 4. The measurement law (under speculative decoding this is absolute)

**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
mapping; chunk counting reported 8.3 tok/s for a config whose server counters
said 23–26 — a 3x false regression that nearly abandoned the change that
worked. Always:

```python
decode_tok_s = usage.completion_tokens / (wall - ttft)
# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
```

vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
nearly shipped the wrong setting.

## 5. Honest methodology

- **Match the corpus to the workload.** Token soup under-reports speculative
  decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
  settings; acceptance, not step time, moves throughput. Build the corpus
  from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
  `--custom-output-len`.
- **Salt every run** with a unique per-arm token in the first bytes — it is
  the prefix-cache key (and a proxy routing key). Without it arm 2 measures
  arm 1's warm cache.
- **Two workload shapes need two clients:** distinct prompts (batch
  behaviour, prefill parallelism) and sequential turns with growing history
  (TTFT of the first streamed chunk per turn — the only measurement that
  exposes prefix-cache behaviour). These pull in opposite directions: a cold
  batch spreads prefill across replicas, a conversation's next turn only pays
  if it lands where its prefix is cached.
- **Read the engine's ledger, not the card:** `Available KV cache memory` /
  `GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
  request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
  is a LIFETIME average across every prompt the replica ever saw — send the
  same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
  → 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
  full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
  chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
  three points before treating a knob as a lever.
- **Soak before believing:** 15–30 min of concurrent families, count failures
  AND latency; report hit/miss split and medians with min/max — the median of
  a bimodal distribution describes neither mode.
- **Verification that is not the benchmark:** run the client the fleet
  actually uses; and a deployed unit is not proven until an unattended reboot
  brings it back.
- **Record the "no" answers** with numbers and mechanism — placement balance,
  concurrency admission and chunk-size tweaks all measured flat or worse
  here; the wall was arithmetic (cached prefixes + live sequences share one
  pool).

## 6. Sweep hygiene (the arms that produced wrong verdicts)

- **Prefetch uniformity:** every arm in a sweep carries the same
  expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
  prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
  12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
  verdict.
- **Knob-landed preflight:** for every `-e` knob, confirm
  `podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
  produces a "parity" arm that is really the default (v1 of the gather sweep
  lost all 3 arms this way; serve-script hooks are additive and can drop
  vars).
- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
  deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
  compare against default-off, not "incumbent + one flag".
- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
  autofit (remainder) so pool numbers are reproducible; compute need =
  tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
  states — measure, don't assume 2x).
- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
  deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
  one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
  `mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.

## 7. Bash laws that ate windows (each cost an arm)

1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
   `SPEC=3: command not found`, nothing runs. Pass literal assignments:
   `SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
   an if/else it aborts ONLY that compound statement — the script CONTINUES,
   every arm tears down clean with zero measurements. Detection law: `ls`
   the per-arm `.measure.txt` files before trusting any summary; arm echoes
   are not evidence.
3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
   it aborts AFTER the harness stopped production. Two statements.
4. `sed -i` renames the file; already-running bash keeps the OLD inode —
   mid-flight patches miss arms launched in the preceding minutes. Patch
   early or run a make-up window.
5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
   (`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
   fails silently. Same trap in `git commit -m`: backticks become command
   substitution.
6. No comments INSIDE a backslash-continued command (the continuation joins
   the comment); verify multi-line command construction with `bash -n` AND a
   render test.

## 8. Results location and recovery (frank-primary arms)

| copy | path |
|---|---|
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |

`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
constants 99.9/88.8 identical across arms by construction; **identical
numbers across arms == stub or bug, never a measurement.** After compaction,
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
window.

## Related

- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
- [[LLM Performance Diagnosis]] — profiling and attribution methods
- [[Remote Job Runner]] — detached execution of long sweeps
- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history