Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
272 lines
15 KiB
Markdown
272 lines
15 KiB
Markdown
---
|
||
type: reference
|
||
title: LLM Bench Windows and Measurement
|
||
description: "Opening and restoring benchmark windows on fleet hosts, honest methodology, and the usage-counter law under drafting."
|
||
tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# LLM Bench Windows and Measurement
|
||
|
||
Fleet law first: **STOP the production service before any GPU experiment**
|
||
(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
|
||
GPU and corrupts results; an unisolated GPU experiment has precedent for
|
||
wedging a host until power cycle. Reference harnesses:
|
||
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
|
||
`scripts/autoresearch/frank-archon-arms.sh`.
|
||
|
||
## 1. The window structure (one config per arm, a driver owns the window)
|
||
|
||
1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
|
||
no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
|
||
EXITED containers and can never clear — `rm -f` the exited one first) AND
|
||
port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
|
||
`ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
|
||
then polls `/health` gets **200 from the still-running PRODUCTION engine**
|
||
— the arm "passes" with production's boot log, then `podman exec` dies
|
||
with `container state improper`.
|
||
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
|
||
LLM hosts — one card, one server). The serve script takes its config via
|
||
env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`)
|
||
pointing at a staged copy; the checkout is never mutated.
|
||
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
|
||
the loop — a refused boot dies in one poll instead of 20 min. Capture the
|
||
serve script's FULL stdout on failure (a `tail -8` loses engine-init root
|
||
causes; raise happens above the tail).
|
||
4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
|
||
SpecDecoding lines) — they survive measure crashes.
|
||
5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
|
||
`podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
|
||
before the next boot; restore production from the EXIT trap and verify the
|
||
**conjunction oracle**: health 200 + `is-active` + `is-enabled` +
|
||
mask-symlink clean. A bare `systemctl start` is not a restore oracle.
|
||
(`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
|
||
enough: systemd never auto-restarts an explicitly stopped unit.)
|
||
6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
|
||
mixed speeds.
|
||
|
||
**Destructive tails (2026-10-01):** a scratch script whose tail does
|
||
`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
|
||
probe loop survived the driver kill — it can destroy a just-deployed
|
||
production engine (signature: deploy exits green, then seconds-to-minutes
|
||
later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
|
||
crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
|
||
and kill strays by explicit PID; killing the dev-side job does NOT kill the
|
||
host-side `deploy-remote.sh` (two of them once managed the same unit).
|
||
|
||
Two failure shapes inside the window that read as "the bench died":
|
||
|
||
- **Stale reads from a stopping container.** `podman stop` answers while the
|
||
engine inside is still flushing; a `podman logs` taken in that window
|
||
returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
|
||
`podman wait`) before reading logs, or you will write "no results" into
|
||
the table for a run that has them.
|
||
- **A D-state worker wedges the stop.** A bench child that wedged in the
|
||
kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
|
||
then the container is `Stopping` forever. Kill by container first
|
||
(`podman kill <ctr>` → SIGKILL path), verify the process is gone from
|
||
`podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
|
||
window script proceed to a redeploy with a zombie bench worker holding
|
||
VRAM.
|
||
|
||
## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
|
||
|
||
**What llama-bench cannot express — decide BEFORE designing:** no
|
||
`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
|
||
activation buffers fit beside the DEPLOYED KV — only the real unit answers
|
||
that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
|
||
(`tg` numbers are the drafterless baseline, useful as a control only), and a
|
||
warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
|
||
cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
|
||
~zero depth and meaningless for long conversations.
|
||
|
||
**The separator trap:** `/` separates devices WITHIN one configuration, `,`
|
||
separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
|
||
single value 36 → everything on one card. `--tensor-split 36,29` (server) →
|
||
`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
|
||
|
||
**Harness shape:** one invocation per cell (an infeasible cell calls
|
||
`exit(1)` and kills every later cell of a cross-product invocation — and
|
||
infeasibility is usually the question); TSV row per cell as it completes +
|
||
`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
|
||
--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
|
||
intended looks fast); resume by skipping non-empty cell outputs; stop the
|
||
service in the harness, restore from `trap … EXIT`, verify afterwards. Match
|
||
the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
|
||
|
||
**Run under a transient unit, not a shell:**
|
||
|
||
```bash
|
||
sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
|
||
--property=LimitMEMLOCK=infinity --property=User=dank \
|
||
/home/dank/serverAppConfigs/scripts/<harness>.sh
|
||
```
|
||
|
||
Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
|
||
8 MiB and fails the load for an unrelated reason. This also survives ssh
|
||
disconnects. **Boot rungs under the production environment, not a bench
|
||
shell** — a rootless shell probe booted 3×262144 bf16 that production refused
|
||
(systemd `User=` + memlock change what the pinned-expert path holds):
|
||
`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
|
||
Restart=no env VARFILE=… /path/serve.sh`.
|
||
|
||
**Verify before burning hours:** dry-run against a STUB binary with
|
||
`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
|
||
including one forced-`exit 3` cell — prove the TSV row, failure capture,
|
||
resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
|
||
nonexistent file. **Reading:** bracket the sweep (reference cell first AND
|
||
last); check per-cell `stddev_ts` (a difference smaller than spread is not a
|
||
result); single-sequence decode is usually flat across batch/ubatch (prefill
|
||
knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
|
||
smaller-batch cells); report what the grid cannot prove (drafter, slots, real
|
||
context).
|
||
|
||
## 3. HTTP benches of server-side parameters
|
||
|
||
`llama-bench` benches the bench binary, not the running server — server-side
|
||
params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
|
||
HTTP:
|
||
|
||
- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
|
||
`prompt_tokens / wall_time`.
|
||
- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
|
||
spec acceptance all stabilize over thousands of tokens); measure
|
||
`predicted_per_second` from `timings` or `completion_tokens / wall_time`;
|
||
temperature 0.
|
||
|
||
Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
|
||
`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
|
||
sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
|
||
reasoning parser streams zero deltas — use `/v1/completions`.
|
||
|
||
## 4. The measurement law (under speculative decoding this is absolute)
|
||
|
||
**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
|
||
mapping; chunk counting reported 8.3 tok/s for a config whose server counters
|
||
said 23–26 — a 3x false regression that nearly abandoned the change that
|
||
worked. Always:
|
||
|
||
```python
|
||
decode_tok_s = usage.completion_tokens / (wall - ttft)
|
||
# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
|
||
```
|
||
|
||
vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
|
||
measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
|
||
them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
|
||
acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
|
||
temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
|
||
(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
|
||
repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
|
||
before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
|
||
nearly shipped the wrong setting.
|
||
|
||
## 5. Honest methodology
|
||
|
||
- **Match the corpus to the workload.** Token soup under-reports speculative
|
||
decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
|
||
settings; acceptance, not step time, moves throughput. Build the corpus
|
||
from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
|
||
`--custom-output-len`.
|
||
- **Salt every run** with a unique per-arm token in the first bytes — it is
|
||
the prefix-cache key (and a proxy routing key). Without it arm 2 measures
|
||
arm 1's warm cache.
|
||
- **Two workload shapes need two clients:** distinct prompts (batch
|
||
behaviour, prefill parallelism) and sequential turns with growing history
|
||
(TTFT of the first streamed chunk per turn — the only measurement that
|
||
exposes prefix-cache behaviour). These pull in opposite directions: a cold
|
||
batch spreads prefill across replicas, a conversation's next turn only pays
|
||
if it lands where its prefix is cached.
|
||
- **Read the engine's ledger, not the card:** `Available KV cache memory` /
|
||
`GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
|
||
request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
|
||
is a LIFETIME average across every prompt the replica ever saw — send the
|
||
same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
|
||
→ 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
|
||
full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
|
||
chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
|
||
three points before treating a knob as a lever.
|
||
- **Soak before believing:** 15–30 min of concurrent families, count failures
|
||
AND latency; report hit/miss split and medians with min/max — the median of
|
||
a bimodal distribution describes neither mode.
|
||
- **Verification that is not the benchmark:** run the client the fleet
|
||
actually uses; and a deployed unit is not proven until an unattended reboot
|
||
brings it back.
|
||
- **Record the "no" answers** with numbers and mechanism — placement balance,
|
||
concurrency admission and chunk-size tweaks all measured flat or worse
|
||
here; the wall was arithmetic (cached prefixes + live sequences share one
|
||
pool).
|
||
|
||
## 6. Sweep hygiene (the arms that produced wrong verdicts)
|
||
|
||
- **Prefetch uniformity:** every arm in a sweep carries the same
|
||
expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
|
||
prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
|
||
12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
|
||
verdict.
|
||
- **Knob-landed preflight:** for every `-e` knob, confirm
|
||
`podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
|
||
produces a "parity" arm that is really the default (v1 of the gather sweep
|
||
lost all 3 arms this way; serve-script hooks are additive and can drop
|
||
vars).
|
||
- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
|
||
deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
|
||
compare against default-off, not "incumbent + one flag".
|
||
- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
|
||
autofit (remainder) so pool numbers are reproducible; compute need =
|
||
tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
|
||
states — measure, don't assume 2x).
|
||
- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
|
||
deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
|
||
one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
|
||
- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
|
||
`mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
|
||
|
||
## 7. Bash laws that ate windows (each cost an arm)
|
||
|
||
1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
|
||
`SPEC=3: command not found`, nothing runs. Pass literal assignments:
|
||
`SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
|
||
2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
|
||
an if/else it aborts ONLY that compound statement — the script CONTINUES,
|
||
every arm tears down clean with zero measurements. Detection law: `ls`
|
||
the per-arm `.measure.txt` files before trusting any summary; arm echoes
|
||
are not evidence.
|
||
3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
|
||
it aborts AFTER the harness stopped production. Two statements.
|
||
4. `sed -i` renames the file; already-running bash keeps the OLD inode —
|
||
mid-flight patches miss arms launched in the preceding minutes. Patch
|
||
early or run a make-up window.
|
||
5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
|
||
(`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
|
||
fails silently. Same trap in `git commit -m`: backticks become command
|
||
substitution.
|
||
6. No comments INSIDE a backslash-continued command (the continuation joins
|
||
the comment); verify multi-line command construction with `bash -n` AND a
|
||
render test.
|
||
|
||
## 8. Results location and recovery (frank-archon arms)
|
||
|
||
| copy | path |
|
||
|---|---|
|
||
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
|
||
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
|
||
| server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` |
|
||
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
|
||
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
|
||
|
||
`/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests —
|
||
constants 99.9/88.8 identical across arms by construction; **identical
|
||
numbers across arms == stub or bug, never a measurement.** After compaction,
|
||
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
|
||
logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
|
||
window.
|
||
|
||
## Related
|
||
|
||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
|
||
- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
|
||
- [[LLM Performance Diagnosis]] — profiling and attribution methods
|
||
- [[Remote Job Runner]] — detached execution of long sweeps
|
||
- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history
|