---
name: llama-bench-fleet-sweep
description: "Run a llama-bench sweep (batch/ubatch, ncmoe, split, or any dimension) on a Dank-fleet LLM host correctly: stop the service, run from a transient unit, one invocation per cell, sample VRAM, restore and verify — including the llama-bench vs llama-server traps that have already produced wrong numbers here."
---

# llama-bench sweeps on a Dank-fleet LLM host

Use when tuning a llama.cpp service's serve-time parameters (batch/ubatch, `-ncmoe`, `-sm`/`-ts`, KV type) offline, i.e. any question of the form "how large may X be / what does X buy".

Reference implementation: `scripts/bench-batch-ubatch.sh` (repo) — read it before writing another harness.

## Preconditions

1. **Stop the service** — Benchmarking Policy in AGENTS.md is mandatory: a bench beside a live service starves the same GPU and corrupts results. Never bench against the live port.
2. **Run from a transient unit, not a shell:**

   ```bash
   sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
        --property=LimitMEMLOCK=infinity --property=User=dank \
        /home/dank/serverAppConfigs/scripts/<harness>.sh
   ```

   The deploy's unit sets `LimitMEMLOCK=infinity` for the ROCm staging buffers; an SSH shell gets 8 MiB (`ulimit -l`), which fails the load for a reason unrelated to what you are measuring. Passwordless sudo is available on both hosts.

## The traps (each one has produced a wrong number)

- **`/` vs `,`.** llama-bench splits `/` = devices *within one configuration*, `,` = *sweep values*. A legitimate llama-server `--tensor-split 36,29` becomes `-ts 36/29`; the comma form silently benches each card alone. Same for `--device ROCm0,ROCm1` → `-dev ROCm0/ROCm1`.
- **No `-c`.** `n_ctx = n_prompt + n_gen + n_depth`. The context is a consequence of the test shape, so a bench can NEVER prove that a configuration's activation buffers fit beside the *deployed* KV — that requires starting the real service at the real context and slot count.
- **A test that cannot allocate calls `exit(1)`**, killing the whole process. One invocation per cell, or one infeasible value loses every later cell — and infeasibility is usually what you are looking for.
- **No `--spec-*` and no `--parallel`.** Decode figures are the *drafterless* baseline, and slots are never exercised. `-d <n>` is what puts KV in front of the measurement: a `tg` instance runs with `n_prompt = 0`, i.e. at ~zero depth otherwise.
- **A cell that looks fast because its allocation was smaller than intended is indistinguishable from a fast cell** unless you sample VRAM. Sample `rocm-smi --showmeminfo vram --csv` every 2 s for the cell's whole life and report min-free per card.
- **`local a=$1 b=$x` expands `$x` before the builtin assigns anything** — under `set -u` that is `b: unbound variable`, and it fires *after* your harness has already stopped the production service. Declare in two statements.
- **Ranges exist:** `-ub 512-2048+256` == `512,768,...,2048`.

## Shape of a good harness

- One `llama-bench` invocation per cell; append one TSV row per cell as it completes, so progress is readable mid-run and an interrupted sweep resumes (skip cells whose `bench.json` is non-empty).
- Write `bench.json` per cell (`-o json`), plus `bench.log` (`--progress`), plus the VRAM trace.
- **Stop the service in the harness and restore it from a `trap ... EXIT`** — and still verify the service afterwards. A harness that crashes mid-run leaves production down.
- Match the deployment's load flags exactly (`-ngl`, `-sm`, `-ts`, `-fa`, `-ctk/-ctv`, `-t`), or you are benching a different service. Note `-ts`/`-dev` take slashes here while the unit takes commas.

## Verify before burning hours

Dry-run the harness against a **stub** `llama-bench` (`BIN=/tmp/stubdir`, plus `SERVICE=<nonexistent-name>` so the real service is untouched) with a tiny grid, including one cell forced to `exit 3`: prove the TSV row, the failure capture, and the resume logic before the real sweep. Then validate the real flag set in ~1 s by pointing `-m` at a nonexistent file — argument errors surface *before* the model load.

## Interpreting results

- A prefill difference smaller than the run-to-run spread is not a result; check each cell's own `stddev_ts`.
- Single-sequence decode is usually flat across batch/ubatch — that knob is prefill-side.
- Batch ≥ ubatch always (llama.cpp clamps ubatch to batch, so smaller-batch cells silently duplicate).
- Report what the harness cannot see (drafter, slots, real context) rather than implying the sweep settled it.