---
name: llama-bench-host-sweep
description: "Run a llama-bench sweep (batch/ubatch, split, quant, depth) on a live fleet LLM host without fooling yourself: what llama-bench cannot express, the separator trap, per-cell isolation, the memlock/transient-unit requirement, and the dry-run that must precede a multi-hour run."
---

# Running a llama-bench sweep on a fleet host

Use when a bench must answer a serving question on one of the LLM hosts (Dankclaw
frank-large/frank-small, Dankserver frank-medium) — batch/ubatch sizing, split
choice, KV type, depth behaviour.

Repo policy applies: **stop the production service first** (AGENTS.md Benchmarking
Policy). The harness below does that for you and restores it.

## 1. What llama-bench cannot express — decide this BEFORE designing the sweep

| Not available | Consequence |
|---|---|
| `-c/--ctx-size` | `n_ctx = n_prompt + n_gen + n_depth` (llama-bench.cpp). The context is a consequence of the test shape, so **the deployment's real allocation is never tested** (e.g. a 16 GiB KV at 262144 over 2 slots). Only starting the real unit answers "does it fit". |
| `--parallel` | single sequence only; no slot shape, no concurrency effect. |
| `--spec-*` | no drafter. Any `tg` number is the **autoregressive baseline**, not the served rate. Useful as a control, never as the deployed figure. |
| warmup per config | a warmup run is itself a full-size run; at 64k prompts `--no-warmup` buys a second sample for the same wall clock. |

`-d N` is a depth PREFIX: it puts N tokens of KV in place before the measured run and
is cached across reps through a `llama_state_seq_*` save/restore, so the fill is paid
once per cell and is not part of the measured number. Without `-d`, a `tg` test runs at
~zero depth and the number is meaningless for long conversations.

## 2. Two traps that silently change the experiment

- **Separators.** `/` separates devices *within one configuration*; `,` separates
  configurations to sweep. `-ts` and `-dev` both follow this. `-ts 36,29` parses as the
  single value `36` → everything lands on one card and you have benched a different
  machine. llama-server takes the comma form for the same pair. List-valued args also
  accept ranges: `-ub 512-2048+256`.
- **memlock.** The deploy units set `LimitMEMLOCK=infinity` because the ROCm backend
  pins staging buffers; a plain SSH shell on these hosts has soft=hard=8 MiB
  (`ulimit -l`), so a load can fail for a reason unrelated to the model. Run the sweep
  inside a transient unit:

  ```bash
  sudo systemd-run --unit=fm-bench --collect --no-block \
      --property=Type=oneshot --property=LimitMEMLOCK=infinity --property=User=dank \
      /home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
  ```

  This also survives an ssh disconnect, unlike a foreground remote process.

## 3. One invocation per cell

A test that cannot run calls `exit(1)` immediately (failed depth/prompt/gen run), so a
single cross-product invocation loses **every later cell** the moment one ubatch fails to
allocate — and infeasibility is usually one of the questions being asked.

Drive the grid from a shell loop: one llama-bench call per (b, ub), continue on a
non-zero exit, record why, and skip cells whose output file already exists and is
non-empty (that makes an interrupted sweep resumable). Write rows to a TSV as the sweep
proceeds so progress is readable mid-run.

## 4. Sample VRAM per cell, covering the load

Batch/ubatch sizes the activation buffers, so a cell that looks fast because its
allocation was silently smaller must be distinguishable from one that is genuinely fast.
Poll `rocm-smi --showmeminfo vram --csv` every 2 s for the cell's whole life and keep the
peak (minimum free) per card. Correlate by taking the window between the cell's start and
end.

## 5. Always restore the service from a trap

`trap <restart> EXIT` in the harness, plus a manual check afterwards. A harness that
stops a production unit and then dies leaves inference down; this fleet has had that
happen.

## 6. Dry-run before a multi-hour run

Stub the binary so the whole path is exercised without loading a 29 GiB model:

```bash
# /tmp/benchstub/llama-bench prints the -o json shape and exits non-zero for one ubatch
SERVICE=frank-medium-dryrun BIN=/tmp/benchstub MODEL=/tmp/benchstub/m.gguf OUT=/tmp/bub-test \
    N_PROMPT=8 N_GEN=4 N_DEPTH=0 REPS=1 BATCHES="2048 4096" UBATCHES="512 1024" \
    bash scripts/bench-batch-ubatch.sh
```

`SERVICE=<nonexistent>` means the trap and the stop are never reached, so the real
service is untouched. Check that a FAILED cell still produces a row with the right column
count before trusting the table.

## 7. Read the result sanely

- Bracket the sweep: run the reference cell (the deployed configuration) first and
  again last, so drift over hours is visible.
- Sanity-check one number against the runbook's prior figures before believing the grid.
- Remember what the grid does NOT prove: it did not run the drafter, the slots, or the
  deployment's context. The winner still has to be applied and verified live.

## Pitfall while writing such a harness

`local b=$1 ub=$2 dir="$OUT/b${b}_ub${ub}"` expands **all** the builtin's arguments
before the builtin assigns any of them, so under `set -u` this aborts with
`b: unbound variable`. Declare in two statements:

```bash
local b=$1 ub=$2
local dir="$OUT/b${b}_ub${ub}" sampler start rc wall
```

Ref: `scripts/bench-batch-ubatch.sh` in serverAppConfigs (the fleet's implementation of
all of the above).