command-center/agent/managed-skills/llama-bench-host-sweep/SKILL.md

name, description

name	description
llama-bench-host-sweep	Run a llama-bench sweep (batch/ubatch, split, quant, depth) on a live fleet LLM host without fooling yourself: what llama-bench cannot express, the separator trap, per-cell isolation, the memlock/transient-unit requirement, and the dry-run that must precede a multi-hour run.

Running a llama-bench sweep on a fleet host

Use when a bench must answer a serving question on one of the LLM hosts (Dankclaw frank-large/frank-small, Dankserver frank-medium) — batch/ubatch sizing, split choice, KV type, depth behaviour.

Repo policy applies: stop the production service first (AGENTS.md Benchmarking Policy). The harness below does that for you and restores it.

1. What llama-bench cannot express — decide this BEFORE designing the sweep

Not available	Consequence
-c/--ctx-size	n_ctx = n_prompt + n_gen + n_depth (llama-bench.cpp). The context is a consequence of the test shape, so the deployment's real allocation is never tested (e.g. a 16 GiB KV at 262144 over 2 slots). Only starting the real unit answers "does it fit".
--parallel	single sequence only; no slot shape, no concurrency effect.
--spec-*	no drafter. Any tg number is the autoregressive baseline, not the served rate. Useful as a control, never as the deployed figure.
warmup per config	a warmup run is itself a full-size run; at 64k prompts --no-warmup buys a second sample for the same wall clock.

-d N is a depth PREFIX: it puts N tokens of KV in place before the measured run and is cached across reps through a llama_state_seq_* save/restore, so the fill is paid once per cell and is not part of the measured number. Without -d, a tg test runs at ~zero depth and the number is meaningless for long conversations.

2. Two traps that silently change the experiment

• Separators. / separates devices within one configuration; , separates configurations to sweep. -ts and -dev both follow this. -ts 36,29 parses as the single value 36 → everything lands on one card and you have benched a different machine. llama-server takes the comma form for the same pair. List-valued args also accept ranges: -ub 512-2048+256.

• memlock. The deploy units set LimitMEMLOCK=infinity because the ROCm backend pins staging buffers; a plain SSH shell on these hosts has soft=hard=8 MiB (ulimit -l), so a load can fail for a reason unrelated to the model. Run the sweep inside a transient unit:

  sudo systemd-run --unit=fm-bench --collect --no-block \
      --property=Type=oneshot --property=LimitMEMLOCK=infinity --property=User=dank \
      /home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
  

  This also survives an ssh disconnect, unlike a foreground remote process.

3. One invocation per cell

A test that cannot run calls exit(1) immediately (failed depth/prompt/gen run), so a single cross-product invocation loses every later cell the moment one ubatch fails to allocate — and infeasibility is usually one of the questions being asked.

Drive the grid from a shell loop: one llama-bench call per (b, ub), continue on a non-zero exit, record why, and skip cells whose output file already exists and is non-empty (that makes an interrupted sweep resumable). Write rows to a TSV as the sweep proceeds so progress is readable mid-run.

4. Sample VRAM per cell, covering the load

Batch/ubatch sizes the activation buffers, so a cell that looks fast because its allocation was silently smaller must be distinguishable from one that is genuinely fast. Poll rocm-smi --showmeminfo vram --csv every 2 s for the cell's whole life and keep the peak (minimum free) per card. Correlate by taking the window between the cell's start and end.

5. Always restore the service from a trap

trap <restart> EXIT in the harness, plus a manual check afterwards. A harness that stops a production unit and then dies leaves inference down; this fleet has had that happen.

6. Dry-run before a multi-hour run

Stub the binary so the whole path is exercised without loading a 29 GiB model:

# /tmp/benchstub/llama-bench prints the -o json shape and exits non-zero for one ubatch
SERVICE=frank-medium-dryrun BIN=/tmp/benchstub MODEL=/tmp/benchstub/m.gguf OUT=/tmp/bub-test \
    N_PROMPT=8 N_GEN=4 N_DEPTH=0 REPS=1 BATCHES="2048 4096" UBATCHES="512 1024" \
    bash scripts/bench-batch-ubatch.sh

SERVICE=<nonexistent> means the trap and the stop are never reached, so the real service is untouched. Check that a FAILED cell still produces a row with the right column count before trusting the table.

7. Read the result sanely

• Bracket the sweep: run the reference cell (the deployed configuration) first and again last, so drift over hours is visible.
• Sanity-check one number against the runbook's prior figures before believing the grid.
• Remember what the grid does NOT prove: it did not run the drafter, the slots, or the deployment's context. The winner still has to be applied and verified live.

Pitfall while writing such a harness

local b=$1 ub=$2 dir="$OUT/b${b}_ub${ub}" expands all the builtin's arguments before the builtin assigns any of them, so under set -u this aborts with b: unbound variable. Declare in two statements:

local b=$1 ub=$2
local dir="$OUT/b${b}_ub${ub}" sampler start rc wall

Ref: scripts/bench-batch-ubatch.sh in serverAppConfigs (the fleet's implementation of all of the above).