serverAppConfigs/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	LLM Bench Windows and Measurement	Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.	benchmarking llama-bench ab-testing speculative vllm fleet	2026-10-05T00:00:00-07:00

LLM Bench Windows and Measurement

Fleet law first: STOP the production service before any GPU experiment (AGENTS.md Benchmarking Policy) — a bench beside a live service starves the GPU and corrupts results; an unisolated GPU experiment has precedent for wedging a host until power cycle. Reference harnesses: scripts/bench-batch-ubatch.sh, scripts/autoresearch.sh, scripts/autoresearch/frank-primary-arms.sh.

1. The window structure (one config per arm, a driver owns the window)

1. Stop + gate before every arm — wait_for_clear: unit NOT active AND no RUNNING container of that name (podman ps, not -a; -a counts EXITED containers and can never clear — rm -f the exited one first) AND port free (ss -ltn). The graceful-stop race cost a whole arm 2026-10-01: ExecStop=podman stop -t 120 drains up to 2 min; a driver that sleeps 3 s then polls /health gets 200 from the still-running PRODUCTION engine — the arm "passes" with production's boot log, then podman exec dies with container state improper.
2. Boot the arm as the service, not a scratch port (no scratch ports on LLM hosts — one card, one server). The serve script takes its config via env-file override (FRANK_SECONDARY_ENV_FILE, FRANK_PRIMARY_AR_ENV_FILE) pointing at a staged copy; the checkout is never mutated.
3. Fast-fail the health wait: poll podman ps | grep -qx <name> inside the loop — a refused boot dies in one poll instead of 20 min. Capture the serve script's FULL stdout on failure (a tail -8 loses engine-init root causes; raise happens above the tail).
4. Measure (§4), capture boot-log engine facts (KV pool, arena slots, SpecDecoding lines) — they survive measure crashes.
5. Teardown + converge: kill by explicit PID/pgid (never pattern kills), podman rm -f scratch, wait MemAvailable ≥ ~230 GiB on dankserver before the next boot; restore production from the EXIT trap and verify the conjunction oracle: health 200 + is-active + is-enabled + mask-symlink clean. A bare systemctl start is not a restore oracle. (systemctl mask FAILS on a relinked unit — log it, explicit stop is enough: systemd never auto-restarts an explicitly stopped unit.)
6. Warm boots cycle ~2 min (page cache), cold ~8 min — size timeouts for mixed speeds.

Destructive tails (2026-10-01): a scratch script whose tail does systemctl stop + podman rm -f <PRODUCTION container> fires later if a probe loop survived the driver kill — it can destroy a just-deployed production engine (signature: deploy exits green, then seconds-to-minutes later Exited (0), Deactivated successfully, clean exit = removed, not crashed). Before ANY deploy: pgrep -af every scratch script of the session and kill strays by explicit PID; killing the dev-side job does NOT kill the host-side deploy-remote.sh (two of them once managed the same unit).

Two failure shapes inside the window that read as "the bench died":

• Stale reads from a stopping container. podman stop answers while the engine inside is still flushing; a podman logs taken in that window returns the boot-prefix only. Poll podman ps for Exited (0) (or podman wait) before reading logs, or you will write "no results" into the table for a run that has them.
• A D-state worker wedges the stop. A bench child that wedged in the kernel ignores SIGTERM, so podman stop -t N burns its full timeout and then the container is Stopping forever. Kill by container first (podman kill <ctr> → SIGKILL path), verify the process is gone from podman ps AND nvidia-smi, and only then re-serve. Never let the window script proceed to a redeploy with a zombie bench worker holding VRAM.

2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)

What llama-bench cannot express — decide BEFORE designing: no --ctx-size (n_ctx = n_prompt + n_gen + n_depth, so a bench can NEVER prove activation buffers fit beside the DEPLOYED KV — only the real unit answers that), no --parallel (no slot shape, no concurrency), no --spec-* (tg numbers are the drafterless baseline, useful as a control only), and a warmup run is itself a full-size run. -d N is a depth PREFIX (paid once per cell via a llama_state_seq_* save/restore); without it a tg run is at ~zero depth and meaningless for long conversations.

The separator trap: / separates devices WITHIN one configuration, , separates sweep values — for both -dev and -ts. -ts 36,29 parses as the single value 36 → everything on one card. --tensor-split 36,29 (server) → -ts 36/29 (bench). Ranges exist: -ub 512-2048+256.

Harness shape: one invocation per cell (an infeasible cell calls exit(1) and kills every later cell of a cross-product invocation — and infeasibility is usually the question); TSV row per cell as it completes + bench.json (-o json) + bench.log (--progress) + a 2 s rocm-smi --showmeminfo vram VRAM trace per cell (a cell that allocated less than intended looks fast); resume by skipping non-empty cell outputs; stop the service in the harness, restore from trap … EXIT, verify afterwards. Match the deployment's load flags exactly (-ngl -sm -ts -fa -ctk/-ctv -t).

Run under a transient unit, not a shell:

sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
     --property=LimitMEMLOCK=infinity --property=User=dank \
     /home/dank/serverAppConfigs/scripts/<harness>.sh

Deploy units set LimitMEMLOCK=infinity for ROCm staging; an SSH shell has 8 MiB and fails the load for an unrelated reason. This also survives ssh disconnects. Boot rungs under the production environment, not a bench shell — a rootless shell probe booted 3×262144 bf16 that production refused (systemd User= + memlock change what the pinned-expert path holds): sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p Restart=no env VARFILE=… /path/serve.sh.

Verify before burning hours: dry-run against a STUB binary with SERVICE=<nonexistent-name> (the trap/stop never reached), tiny grid, including one forced-exit 3 cell — prove the TSV row, failure capture, resume logic. Validate the real flag set in ~1 s by pointing -m at a nonexistent file. Reading: bracket the sweep (reference cell first AND last); check per-cell stddev_ts (a difference smaller than spread is not a result); single-sequence decode is usually flat across batch/ubatch (prefill knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating smaller-batch cells); report what the grid cannot prove (drafter, slots, real context).

3. HTTP benches of server-side parameters

llama-bench benches the bench binary, not the running server — server-side params (--threads-batch, --ubatch-size, --spec-type, --parallel) need HTTP:

• Prefill: large prompt, max_tokens=1 (server crashes on 0), measure prompt_tokens / wall_time.
• Decode: max_tokens=16384 minimum, NEVER 1 (warmup, cache growth and spec acceptance all stabilize over thousands of tokens); measure predicted_per_second from timings or completion_tokens / wall_time; temperature 0.

Known fleet harnesses: scripts/bench-threads-batch-http.py, scripts/bench-decode-threads-batch.py, scripts/bench-spec-decode.py. sglang-specific: max_tokens=1 on /v1/chat/completions with the qwen3 reasoning parser streams zero deltas — use /v1/completions.

4. The measurement law (under speculative decoding this is absolute)

Never count SSE chunks. Rejected drafts break the 1:1 chunk↔token mapping; chunk counting reported 8.3 tok/s for a config whose server counters said 23–26 — a 3x false regression that nearly abandoned the change that worked. Always:

decode_tok_s = usage.completion_tokens / (wall - ttft)
# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk

vLLM's 10 s Accepted throughput log lines are a rolling window, not a measurement (6.2 → 13.1 across adjacent windows on identical settings) — use them for acceptance STATISTICS only (Avg Draft acceptance rate, Mean acceptance length, per-position rates). Warm pass + ≥3 runs, median; temperature 0; ≥256 tokens. A decode "jump" with early-stopped completions (ct < 256) and acceptance > 90 % is degenerate generation (the fork's repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample nearly shipped the wrong setting.

5. Honest methodology

• Match the corpus to the workload. Token soup under-reports speculative decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft settings; acceptance, not step time, moves throughput. Build the corpus from the repo's own sources: {"prompt": "<instruction>\n\n<file body>"}, --custom-output-len.
• Salt every run with a unique per-arm token in the first bytes — it is the prefix-cache key (and a proxy routing key). Without it arm 2 measures arm 1's warm cache.
• Two workload shapes need two clients: distinct prompts (batch behaviour, prefill parallelism) and sequential turns with growing history (TTFT of the first streamed chunk per turn — the only measurement that exposes prefix-cache behaviour). These pull in opposite directions: a cold batch spreads prefill across replicas, a conversation's next turn only pays if it lands where its prefix is cached.
• Read the engine's ledger, not the card: Available KV cache memory / GPU KV cache size: N tokens / Maximum concurrency for M tokens per request: X.Xx is the capacity number. vLLM's Prefix cache hit rate: line is a LIFETIME average across every prompt the replica ever saw — send the same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold → 1.4 s warm for a 50k prefix). GPU KV cache usage says the pool is not full; it does NOT say a prefix survived. The KV budget is NOT monotonic in chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure three points before treating a knob as a lever.
• Soak before believing: 15–30 min of concurrent families, count failures AND latency; report hit/miss split and medians with min/max — the median of a bimodal distribution describes neither mode.
• Verification that is not the benchmark: run the client the fleet actually uses; and a deployed unit is not proven until an unattended reboot brings it back.
• Record the "no" answers with numbers and mechanism — placement balance, concurrency admission and chunk-size tweaks all measured flat or worse here; the wall was arithmetic (cached prefixes + live sequences share one pool).

6. Sweep hygiene (the arms that produced wrong verdicts)

• Prefetch uniformity: every arm in a sweep carries the same expert-prefetch setting. The 2026-09-29 campaign measured losing arms with prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec 12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed verdict.
• Knob-landed preflight: for every -e knob, confirm podman exec <ctr> printenv <VAR> — a knob the fork silently ignores produces a "parity" arm that is really the default (v1 of the gather sweep lost all 3 arms this way; serve-script hooks are additive and can drop vars).
• Always run an explicit OFF arm: the 2026-09-29 campaign's only win was deleting a baseline's silent --expert-prefetch 1 (+40–51 %). Challengers compare against default-off, not "incumbent + one flag".
• Pin the capacity bar: --kv-cache-memory-bytes (contract) rather than autofit (remainder) so pool numbers are reproducible; compute need = tokens × measured B/token (hybrid models: KV dtype does NOT double mamba states — measure, don't assume 2x).
• Report decode AND prefill (TTFT) per rung: a decode-only view nearly deployed a 2x-prefill-tax config. Per-depth tables, never a single median — one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
• Container recreate (podman rm -f) wipes podman cp'd harnesses — re-copy mtp_measure.py/bench_iter.py after EVERY recreate, or mount them.

7. Bash laws that ate windows (each cost an arm)

1. ${VAR:+SPEC=$1} in front of a command expands to a bare word → SPEC=3: command not found, nothing runs. Pass literal assignments: SPEC="${1:-}" bash serve.sh; the callee's ${SPEC:+--flag} skips empty.
2. ${2WAY:-0} is fatal bad substitution (digit-leading name), and inside an if/else it aborts ONLY that compound statement — the script CONTINUES, every arm tears down clean with zero measurements. Detection law: ls the per-arm .measure.txt files before trusting any summary; arm echoes are not evidence.
3. local a=$1 b=$x expands $x before the builtin assigns — under set -u it aborts AFTER the harness stopped production. Two statements.
4. sed -i renames the file; already-running bash keeps the OLD inode — mid-flight patches miss arms launched in the preceding minutes. Patch early or run a make-up window.
5. Never heredoc python through ssh→zsh→bash — write to a file, ship it (ssh host 'cat > /path && chmod +x /path' < local.sh); nested quoting fails silently. Same trap in git commit -m: backticks become command substitution.
6. No comments INSIDE a backslash-continued command (the continuation joins the comment); verify multi-line command construction with bash -n AND a render test.

8. Results location and recovery (frank-primary arms)

copy	path
per-arm evidence	$ARMS_DIR = ~/.jobs/<run-stamp>/ (.rc, .done beside it)
summary	$ARMS_DIR/summary.tsv = arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s + <arm>.out/.env/.overlay/.rc/.metrics
server-side last window	/home/dank/.autoresearch-frank-primary/run.json
durable memo	.../last/run.json + last/hash (promoted only after bench AND verified restore)
fallback	~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log (permanent)

/tmp/frank-primary-arms-* and /tmp/armstub/ are stub self-tests — constants 99.9/88.8 identical across arms by construction; identical numbers across arms == stub or bug, never a measurement. After compaction, re-harvest with grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+' over the session logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per window.

Related

• RDNA4 MoE Expert Offload VRAM Budget — the plan arithmetic being swept
• DFlash DSpark Speculative Decoding — spec A/B protocol and shapes
• LLM Performance Diagnosis — profiling and attribution methods
• Remote Job Runner — detached execution of long sweeps
• frank-medium-llamacpp-dflash2 — the measured ubatch-optimum history