command-center/agent/managed-skills/llamacpp-layer-split-vram-rebalance/SKILL.md

name, description

name	description
llamacpp-layer-split-vram-rebalance	Re-measure and set the --tensor-split VRAM balance for a multi-GPU llama.cpp server when the batch, context, slots or spec drafter change, instead of interpolating between splits.

Re-balancing is needed whenever a knob moves weight between cards: --ubatch-size (compute buffers), a spec drafter (weights + draft KV), --ctx-size/--parallel (KV), or a different drafter with a different window policy. A split balanced at one configuration is unbalanced at another — and the second card's asymmetry (output layers + drafter graphs) is what makes it non-obvious.

0. Stop the service first

Serving a user? Stop it, then free the port, before loading test instances. Do not bench beside a live server.

sudo systemctl stop <svc>        # root unit; for a rootless quadlet use systemctl --user
pkill -x llama-server && sleep 3 # a manual/child run holds the port and the VRAM

pkill -f <pattern> is forbidden here: it matches your own command line and kills the ssh session. mask FAILS on a unit that is a symlink into a repo checkout (File ... already exists and is a symlink to ...); a stop is enough because nothing else restarts it.

1. Measure one split per load

The serve script is env-driven, so point it at a copy of the env file with TENSOR_SPLIT patched and read the allocation back from rocm-smi. Load -> sample -> kill, with NO requests sent: this touches allocation only, not the compute path.

WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
export FRANK_MEDIUM_ENV_FILE=$WORK          # the script's own override hook
for ts in 39,26 38,27 37,28 36,29 35,30; do
  sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
  nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
  for _ in $(seq 1 150); do
    [ "$(curl -s -o /dev/null -m 2 -w '%{http_code}' http://127.0.0.1:PORT/health)" = 200 ] && break
    pgrep -x llama-server >/dev/null || break
    sleep 2
  done
  python3 -c "
import subprocess,re
CAP=34208743424  # one card, as rocm-smi reports it
o=subprocess.run(['rocm-smi','--showmeminfo','vram'],capture_output=True,text=True).stdout
u=[int(x) for x in re.findall(r'VRAM Total Used Memory \\(B\\): (\\d+)',o)][:2]
print('$ts', f'{(CAP-u[0])/2**30:.2f}', f'{(CAP-u[1])/2**30:.2f}')"
  pkill -x llama-server; sleep 6
done

Sample ~6 s after /health returns 200 — the last allocations land after listening. Run it under nohup and poll the log from a separate call: a load is 30-90 s, which exceeds a single command's patience.

2. Read the table, do not interpolate

Pick the split with the smallest inter-card gap, then confirm it clears any per-card floor (e.g. 1.5 GiB) with room to spare. A -ts step is ONE LAYER, and hybrid stacks do not have uniform layers. Measured on qwen35 (16 x (3 Gated DeltaNet + 1 Gated Attention)): 0.43 GiB on a DeltaNet boundary, 1.43 GiB on a full-attention one (weights + that layer's own f16 KV). So the gap can jump 2.9 GiB in a single step, and the best integer split is ~0.2 GiB from balanced while its neighbours are 1+ GiB off. Interpolating between two measured splits produces a split nobody should deploy.

Total free VRAM is invariant across splits (only its distribution moves) — use that to sanity-check a suspicious row.

3. Check the drafter's window before blaming the split

A draft context is built with n_ctx = llama_n_ctx(ctx_tgt) and there is no flag to cap it, so a drafter declaring NO sliding window caches the WHOLE target window on --spec-draft-device. DFlash2 (sliding_window 2048) = ~40 MiB; DSpark (no window) = 20 KiB/position = 5.0 GiB at 262144 — a four-layer swing in the balance point. Read sliding_window out of the drafter's GGUF header before sizing, and prefer a skewed -ts over cutting --ctx-size.

4. If the repo has a fitted VRAM model, refit it — with the same unit convention

Match the existing expression literally (e.g. 1442 * 1024 * ubatch means ~1.4 MB per ubatch unit, not 1442 bytes). Fit so the deployed point reproduces rocm-smi to ~0.05 GiB, keep the physics (weights + proportional KV + a window-bounded draft KV + ubatch-scaled buffers), and treat a NEGATIVE fitted constant as a unit bug. State in the comment where the model is conservative: a uniform per-layer cost overstates gaps in a hybrid stack, and a fitted ubatch coefficient makes any larger-batch verdict an extrapolation, not a measurement.

5. Verify behaviourally after deploy, not just by flags

• /slots -> the expected (id, n_ctx) pairs.
• The drafter's init lines from journalctl -u <svc>: implementation name, n_max, block_size — read the NUMBERS, since DFlash2 and DSpark log under the same draft_dflash symbol.
• rocm-smi free VRAM at the deployed split (the fitted prediction and the real figure should agree to ~0.05 GiB).
• One tool call and two CONCURRENT requests, one per slot, then journalctl -o cat | grep 'slot print_timing' | grep -o 'slot print_timing: id *[01].*draft acceptance.*' to prove drafting is live on BOTH slots (journalctl default output truncates these lines; -o cat plus grep -o gets them whole).
• grep -c 'ROCm error\|illegal memory\|ggml_abort' on the current instance must be 0.

Faults to watch, not to tune away

A cross-card boundary copy can fault (amdgpu SDMA0 UTCL2 page fault -> ROCm error: an illegal memory access was encountered -> ggml_abort) on a split that loads and answers /health perfectly. It is not an OOM — more headroom does not prevent it — so treat a split that has faulted as off-limits and re-measure rather than re-tuning. Also: a load that succeeds and then dies on the FIRST draft decode with an OOM is a sizing failure (the draft KV), not a broken drafter.