serverAppConfigs/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	LLM Bench Windows and Measurement	Opening and restoring benchmark windows on fleet hosts, honest methodology, and the usage-counter law under drafting.	benchmarking llama-bench ab-testing speculative vllm fleet	2026-10-05T00:00:00-07:00

LLM Bench Windows and Measurement

Fleet law first: STOP the production service before any GPU experiment (AGENTS.md Benchmarking Policy) — a bench beside a live service starves the GPU and corrupts results; an unisolated GPU experiment has precedent for wedging a host until power cycle. Reference harnesses: scripts/bench-batch-ubatch.sh, scripts/autoresearch.sh, scripts/autoresearch/frank-archon-arms.sh.

1. The window structure (one config per arm, a driver owns the window)

1. Stop + gate before every arm — wait_for_clear: unit NOT active AND no RUNNING container of that name (podman ps, not -a; -a counts EXITED containers and can never clear — rm -f the exited one first) AND port free (ss -ltn). The graceful-stop race cost a whole arm 2026-10-01: ExecStop=podman stop -t 120 drains up to 2 min; a driver that sleeps 3 s then polls /health gets 200 from the still-running PRODUCTION engine — the arm "passes" with production's boot log, then podman exec dies with container state improper.
2. Boot the arm as the service, not a scratch port (no scratch ports on LLM hosts — one card, one server). The serve script takes its config via env-file override (FRANK_ZEALOT_ENV_FILE, FRANK_ARCHON_AR_ENV_FILE) pointing at a staged copy; the checkout is never mutated.
3. Fast-fail the health wait: poll podman ps | grep -qx <name> inside the loop — a refused boot dies in one poll instead of 20 min. Capture the serve script's FULL stdout on failure (a tail -8 loses engine-init root causes; raise happens above the tail).
4. Measure (§4), capture boot-log engine facts (KV pool, arena slots, SpecDecoding lines) — they survive measure crashes.
5. Teardown + converge: kill by explicit PID/pgid (never pattern kills), podman rm -f scratch, wait MemAvailable ≥ ~230 GiB on dankserver before the next boot; restore production from the EXIT trap and verify the conjunction oracle: health 200 + is-active + is-enabled + mask-symlink clean. A bare systemctl start is not a restore oracle. (systemctl mask FAILS on a relinked unit — log it, explicit stop is enough: systemd never auto-restarts an explicitly stopped unit.)
6. Warm boots cycle ~2 min (page cache), cold ~8 min — size timeouts for mixed speeds.

Destructive tails (2026-10-01): a scratch script whose tail does systemctl stop + podman rm -f <PRODUCTION container> fires later if a probe loop survived the driver kill — it can destroy a just-deployed production engine (signature: deploy exits green, then seconds-to-minutes later Exited (0), Deactivated successfully, clean exit = removed, not crashed). Before ANY deploy: pgrep -af every scratch script of the session and kill strays by explicit PID; killing the dev-side job does NOT kill the host-side deploy-remote.sh (two of them once managed the same unit).

Two failure shapes inside the window that read as "the bench died":

• Stale reads from a stopping container. podman stop answers while the engine inside is still flushing; a podman logs taken in that window returns the boot-prefix only. Poll podman ps for Exited (0) (or podman wait) before reading logs, or you will write "no results" into the table for a run that has them.
• A D-state worker wedges the stop. A bench child that wedged in the kernel ignores SIGTERM, so podman stop -t N burns its full timeout and then the container is Stopping forever. Kill by container first (podman kill <ctr> → SIGKILL path), verify the process is gone from podman ps AND nvidia-smi, and only then re-serve. Never let the window script proceed to a redeploy with a zombie bench worker holding VRAM.

2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)

What llama-bench cannot express — decide BEFORE designing: no --ctx-size (n_ctx = n_prompt + n_gen + n_depth, so a bench can NEVER prove activation buffers fit beside the DEPLOYED KV — only the real unit answers that), no --parallel (no slot shape, no concurrency), no --spec-* (tg numbers are the drafterless baseline, useful as a control only), and a warmup run is itself a full-size run. -d N is a depth PREFIX (paid once per cell via a llama_state_seq_* save/restore); without it a tg run is at ~zero depth and meaningless for long conversations.

The separator trap: / separates devices WITHIN one configuration, , separates sweep values — for both -dev and -ts. -ts 36,29 parses as the single value 36 → everything on one card. --tensor-split 36,29 (server) → -ts 36/29 (bench). Ranges exist: -ub 512-2048+256.

Harness shape: one invocation per cell (an infeasible cell calls exit(1) and kills every later cell of a cross-product invocation — and infeasibility is usually the question); TSV row per cell as it completes + bench.json (-o json) + bench.log (--progress) + a 2 s rocm-smi --showmeminfo vram VRAM trace per cell (a cell that allocated less than intended looks fast); resume by skipping non-empty cell outputs; stop the service in the harness, restore from trap … EXIT, verify afterwards. Match the deployment's load flags exactly (-ngl -sm -ts -fa -ctk/-ctv -t).

Run under a transient unit, not a shell:

sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
     --property=LimitMEMLOCK=infinity --property=User=dank \
     /home/dank/serverAppConfigs/scripts/<harness>.sh

Deploy units set LimitMEMLOCK=infinity for ROCm staging; an SSH shell has 8 MiB and fails the load for an unrelated reason. This also survives ssh disconnects. Boot rungs under the production environment, not a bench shell — a rootless shell probe booted 3×262144 bf16 that production refused (systemd User= + memlock change what the pinned-expert path holds): sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p Restart=no env VARFILE=… /path/serve.sh.

Verify before burning hours: dry-run against a STUB binary with SERVICE=<nonexistent-name> (the trap/stop never reached), tiny grid, including one forced-exit 3 cell — prove the TSV row, failure capture, resume logic. Validate the real flag set in ~1 s by pointing -m at a nonexistent file. Reading: bracket the sweep (reference cell first AND last); check per-cell stddev_ts (a difference smaller than spread is not a result); single-sequence decode is usually flat across batch/ubatch (prefill knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating smaller-batch cells); report what the grid cannot prove (drafter, slots, real context).

3. HTTP benches of server-side parameters

llama-bench benches the bench binary, not the running server — server-side params (--threads-batch, --ubatch-size, --spec-type, --parallel) need HTTP:

• Prefill: large prompt, max_tokens=1 (server crashes on 0), measure prompt_tokens / wall_time.
• Decode: max_tokens=16384 minimum, NEVER 1 (warmup, cache growth and spec acceptance all stabilize over thousands of tokens); measure predicted_per_second from timings or completion_tokens / wall_time; temperature 0.

Known fleet harnesses: scripts/bench-threads-batch-http.py, scripts/bench-decode-threads-batch.py, scripts/bench-spec-decode.py. sglang-specific: max_tokens=1 on /v1/chat/completions with the qwen3 reasoning parser streams zero deltas — use /v1/completions.

4. The measurement law (under speculative decoding this is absolute)

Never count SSE chunks. Rejected drafts break the 1:1 chunk↔token mapping; chunk counting reported 8.3 tok/s for a config whose server counters said 23–26 — a 3x false regression that nearly abandoned the change that worked. Always:

decode_tok_s = usage.completion_tokens / (wall - ttft)
# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk

vLLM's 10 s Accepted throughput log lines are a rolling window, not a measurement (6.2 → 13.1 across adjacent windows on identical settings) — use them for acceptance STATISTICS only (Avg Draft acceptance rate, Mean acceptance length, per-position rates). Warm pass + ≥3 runs, median; temperature 0; ≥256 tokens. A decode "jump" with early-stopped completions (ct < 256) and acceptance > 90 % is degenerate generation (the fork's repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample nearly shipped the wrong setting.

5. Honest methodology

• Match the corpus to the workload. Token soup under-reports speculative decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft settings; acceptance, not step time, moves throughput. Build the corpus from the repo's own sources: {"prompt": "<instruction>\n\n<file body>"}, --custom-output-len.
• Salt every run with a unique per-arm token in the first bytes — it is the prefix-cache key (and a proxy routing key). Without it arm 2 measures arm 1's warm cache.
• Two workload shapes need two clients: distinct prompts (batch behaviour, prefill parallelism) and sequential turns with growing history (TTFT of the first streamed chunk per turn — the only measurement that exposes prefix-cache behaviour). These pull in opposite directions: a cold batch spreads prefill across replicas, a conversation's next turn only pays if it lands where its prefix is cached.
• Read the engine's ledger, not the card: Available KV cache memory / GPU KV cache size: N tokens / Maximum concurrency for M tokens per request: X.Xx is the capacity number. vLLM's Prefix cache hit rate: line is a LIFETIME average across every prompt the replica ever saw — send the same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold → 1.4 s warm for a 50k prefix). GPU KV cache usage says the pool is not full; it does NOT say a prefix survived. The KV budget is NOT monotonic in chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure three points before treating a knob as a lever.
• Soak before believing: 15–30 min of concurrent families, count failures AND latency; report hit/miss split and medians with min/max — the median of a bimodal distribution describes neither mode.
• Verification that is not the benchmark: run the client the fleet actually uses; and a deployed unit is not proven until an unattended reboot brings it back.
• Record the "no" answers with numbers and mechanism — placement balance, concurrency admission and chunk-size tweaks all measured flat or worse here; the wall was arithmetic (cached prefixes + live sequences share one pool).

6. Sweep hygiene (the arms that produced wrong verdicts)

• Prefetch uniformity: every arm in a sweep carries the same expert-prefetch setting. The 2026-09-29 campaign measured losing arms with prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec 12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed verdict.
• Knob-landed preflight: for every -e knob, confirm podman exec <ctr> printenv <VAR> — a knob the fork silently ignores produces a "parity" arm that is really the default (v1 of the gather sweep lost all 3 arms this way; serve-script hooks are additive and can drop vars).
• Always run an explicit OFF arm: the 2026-09-29 campaign's only win was deleting a baseline's silent --expert-prefetch 1 (+40–51 %). Challengers compare against default-off, not "incumbent + one flag".
• Pin the capacity bar: --kv-cache-memory-bytes (contract) rather than autofit (remainder) so pool numbers are reproducible; compute need = tokens × measured B/token (hybrid models: KV dtype does NOT double mamba states — measure, don't assume 2x).
• Report decode AND prefill (TTFT) per rung: a decode-only view nearly deployed a 2x-prefill-tax config. Per-depth tables, never a single median — one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
• Container recreate (podman rm -f) wipes podman cp'd harnesses — re-copy mtp_measure.py/bench_iter.py after EVERY recreate, or mount them.

7. Bash laws that ate windows (each cost an arm)

1. ${VAR:+SPEC=$1} in front of a command expands to a bare word → SPEC=3: command not found, nothing runs. Pass literal assignments: SPEC="${1:-}" bash serve.sh; the callee's ${SPEC:+--flag} skips empty.
2. ${2WAY:-0} is fatal bad substitution (digit-leading name), and inside an if/else it aborts ONLY that compound statement — the script CONTINUES, every arm tears down clean with zero measurements. Detection law: ls the per-arm .measure.txt files before trusting any summary; arm echoes are not evidence.
3. local a=$1 b=$x expands $x before the builtin assigns — under set -u it aborts AFTER the harness stopped production. Two statements.
4. sed -i renames the file; already-running bash keeps the OLD inode — mid-flight patches miss arms launched in the preceding minutes. Patch early or run a make-up window.
5. Never heredoc python through ssh→zsh→bash — write to a file, ship it (ssh host 'cat > /path && chmod +x /path' < local.sh); nested quoting fails silently. Same trap in git commit -m: backticks become command substitution.
6. No comments INSIDE a backslash-continued command (the continuation joins the comment); verify multi-line command construction with bash -n AND a render test.

8. Results location and recovery (frank-archon arms)

copy	path
per-arm evidence	$ARMS_DIR = ~/.jobs/<run-stamp>/ (.rc, .done beside it)
summary	$ARMS_DIR/summary.tsv = arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s + <arm>.out/.env/.overlay/.rc/.metrics
server-side last window	/home/dank/.autoresearch-frank-archon/run.json
durable memo	.../last/run.json + last/hash (promoted only after bench AND verified restore)
fallback	~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log (permanent)

/tmp/frank-archon-arms-* and /tmp/armstub/ are stub self-tests — constants 99.9/88.8 identical across arms by construction; identical numbers across arms == stub or bug, never a measurement. After compaction, re-harvest with grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+' over the session logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per window.

Related

• RDNA4 MoE Expert Offload VRAM Budget — the plan arithmetic being swept
• DFlash DSpark Speculative Decoding — spec A/B protocol and shapes
• LLM Performance Diagnosis — profiling and attribution methods
• Remote Job Runner — detached execution of long sweeps
• frank-medium-llamacpp-dflash2 — the measured ubatch-optimum history