command-center/agent/managed-skills/llama-bench-fleet-sweep/SKILL.md

name, description

name	description
llama-bench-fleet-sweep	Run a llama-bench sweep (batch/ubatch, ncmoe, split, or any dimension) on a Dank-fleet LLM host correctly: stop the service, run from a transient unit, one invocation per cell, sample VRAM, restore and verify — including the llama-bench vs llama-server traps that have already produced wrong numbers here.

llama-bench sweeps on a Dank-fleet LLM host

Use when tuning a llama.cpp service's serve-time parameters (batch/ubatch, -ncmoe, -sm/-ts, KV type) offline, i.e. any question of the form "how large may X be / what does X buy".

Reference implementation: scripts/bench-batch-ubatch.sh (repo) — read it before writing another harness.

Preconditions

1. Stop the service — Benchmarking Policy in AGENTS.md is mandatory: a bench beside a live service starves the same GPU and corrupts results. Never bench against the live port.

2. Run from a transient unit, not a shell:

   sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
        --property=LimitMEMLOCK=infinity --property=User=dank \
        /home/dank/serverAppConfigs/scripts/<harness>.sh
   

   The deploy's unit sets LimitMEMLOCK=infinity for the ROCm staging buffers; an SSH shell gets 8 MiB (ulimit -l), which fails the load for a reason unrelated to what you are measuring. Passwordless sudo is available on both hosts.

The traps (each one has produced a wrong number)

• / vs ,. llama-bench splits / = devices within one configuration, , = sweep values. A legitimate llama-server --tensor-split 36,29 becomes -ts 36/29; the comma form silently benches each card alone. Same for --device ROCm0,ROCm1 → -dev ROCm0/ROCm1.
• No -c. n_ctx = n_prompt + n_gen + n_depth. The context is a consequence of the test shape, so a bench can NEVER prove that a configuration's activation buffers fit beside the deployed KV — that requires starting the real service at the real context and slot count.
• A test that cannot allocate calls exit(1), killing the whole process. One invocation per cell, or one infeasible value loses every later cell — and infeasibility is usually what you are looking for.
• No --spec-* and no --parallel. Decode figures are the drafterless baseline, and slots are never exercised. -d <n> is what puts KV in front of the measurement: a tg instance runs with n_prompt = 0, i.e. at ~zero depth otherwise.
• A cell that looks fast because its allocation was smaller than intended is indistinguishable from a fast cell unless you sample VRAM. Sample rocm-smi --showmeminfo vram --csv every 2 s for the cell's whole life and report min-free per card.
• local a=$1 b=$x expands $x before the builtin assigns anything — under set -u that is b: unbound variable, and it fires after your harness has already stopped the production service. Declare in two statements.
• Ranges exist: -ub 512-2048+256 == 512,768,...,2048.

Shape of a good harness

• One llama-bench invocation per cell; append one TSV row per cell as it completes, so progress is readable mid-run and an interrupted sweep resumes (skip cells whose bench.json is non-empty).
• Write bench.json per cell (-o json), plus bench.log (--progress), plus the VRAM trace.
• Stop the service in the harness and restore it from a trap ... EXIT — and still verify the service afterwards. A harness that crashes mid-run leaves production down.
• Match the deployment's load flags exactly (-ngl, -sm, -ts, -fa, -ctk/-ctv, -t), or you are benching a different service. Note -ts/-dev take slashes here while the unit takes commas.

Verify before burning hours

Dry-run the harness against a stub llama-bench (BIN=/tmp/stubdir, plus SERVICE=<nonexistent-name> so the real service is untouched) with a tiny grid, including one cell forced to exit 3: prove the TSV row, the failure capture, and the resume logic before the real sweep. Then validate the real flag set in ~1 s by pointing -m at a nonexistent file — argument errors surface before the model load.

Interpreting results

• A prefill difference smaller than the run-to-run spread is not a result; check each cell's own stddev_ts.
• Single-sequence decode is usually flat across batch/ubatch — that knob is prefill-side.
• Batch ≥ ubatch always (llama.cpp clamps ubatch to batch, so smaller-batch cells silently duplicate).
• Report what the harness cannot see (drafter, slots, real context) rather than implying the sweep settled it.