---
name: llamacpp-layer-split-vram-rebalance
description: "Re-measure and set the --tensor-split VRAM balance for a multi-GPU llama.cpp server when the batch, context, slots or spec drafter change, instead of interpolating between splits."
---

Re-balancing is needed whenever a knob moves weight between cards: `--ubatch-size` (compute buffers), a spec drafter (weights + draft KV), `--ctx-size`/`--parallel` (KV), or a different drafter with a different window policy. A split balanced at one configuration is unbalanced at another — and the second card's asymmetry (output layers + drafter graphs) is what makes it non-obvious.

## 0. Stop the service first

Serving a user? Stop it, then free the port, before loading test instances. Do not bench beside a live server.

```bash
sudo systemctl stop <svc>        # root unit; for a rootless quadlet use systemctl --user
pkill -x llama-server && sleep 3 # a manual/child run holds the port and the VRAM
```

`pkill -f <pattern>` is forbidden here: it matches your own command line and kills the ssh session. `mask` FAILS on a unit that is a symlink into a repo checkout (`File ... already exists and is a symlink to ...`); a `stop` is enough because nothing else restarts it.

## 1. Measure one split per load

The serve script is env-driven, so point it at a copy of the env file with `TENSOR_SPLIT` patched and read the allocation back from `rocm-smi`. Load -> sample -> kill, with NO requests sent: this touches allocation only, not the compute path.

```bash
WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
export FRANK_MEDIUM_ENV_FILE=$WORK          # the script's own override hook
for ts in 39,26 38,27 37,28 36,29 35,30; do
  sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
  nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
  for _ in $(seq 1 150); do
    [ "$(curl -s -o /dev/null -m 2 -w '%{http_code}' http://127.0.0.1:PORT/health)" = 200 ] && break
    pgrep -x llama-server >/dev/null || break
    sleep 2
  done
  python3 -c "
import subprocess,re
CAP=34208743424  # one card, as rocm-smi reports it
o=subprocess.run(['rocm-smi','--showmeminfo','vram'],capture_output=True,text=True).stdout
u=[int(x) for x in re.findall(r'VRAM Total Used Memory \\(B\\): (\\d+)',o)][:2]
print('$ts', f'{(CAP-u[0])/2**30:.2f}', f'{(CAP-u[1])/2**30:.2f}')"
  pkill -x llama-server; sleep 6
done
```

Sample ~6 s after `/health` returns 200 — the last allocations land after listening. Run it under `nohup` and poll the log from a separate call: a load is 30-90 s, which exceeds a single command's patience.

## 2. Read the table, do not interpolate

Pick the split with the smallest inter-card gap, then confirm it clears any per-card floor (e.g. 1.5 GiB) with room to spare. **A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.** Measured on qwen35 (16 x (3 Gated DeltaNet + 1 Gated Attention)): 0.43 GiB on a DeltaNet boundary, 1.43 GiB on a full-attention one (weights + that layer's own f16 KV). So the gap can jump 2.9 GiB in a single step, and the best integer split is ~0.2 GiB from balanced while its neighbours are 1+ GiB off. Interpolating between two measured splits produces a split nobody should deploy.

Total free VRAM is invariant across splits (only its distribution moves) — use that to sanity-check a suspicious row.

## 3. Check the drafter's window before blaming the split

A draft context is built with `n_ctx = llama_n_ctx(ctx_tgt)` and there is no flag to cap it, so a drafter declaring NO sliding window caches the WHOLE target window on `--spec-draft-device`. DFlash2 (`sliding_window 2048`) = ~40 MiB; DSpark (no window) = 20 KiB/position = 5.0 GiB at 262144 — a four-layer swing in the balance point. Read `sliding_window` out of the drafter's GGUF header before sizing, and prefer a skewed `-ts` over cutting `--ctx-size`.

## 4. If the repo has a fitted VRAM model, refit it — with the same unit convention

Match the existing expression literally (e.g. `1442 * 1024 * ubatch` means ~1.4 MB per ubatch unit, not 1442 bytes). Fit so the deployed point reproduces `rocm-smi` to ~0.05 GiB, keep the physics (weights + proportional KV + a window-bounded draft KV + ubatch-scaled buffers), and treat a NEGATIVE fitted constant as a unit bug. State in the comment where the model is conservative: a uniform per-layer cost overstates gaps in a hybrid stack, and a fitted ubatch coefficient makes any larger-batch verdict an extrapolation, not a measurement.

## 5. Verify behaviourally after deploy, not just by flags

- `/slots` -> the expected `(id, n_ctx)` pairs.
- The drafter's init lines from `journalctl -u <svc>`: implementation name, `n_max`, `block_size` — read the NUMBERS, since DFlash2 and DSpark log under the same `draft_dflash` symbol.
- `rocm-smi` free VRAM at the deployed split (the fitted prediction and the real figure should agree to ~0.05 GiB).
- One tool call and two CONCURRENT requests, one per slot, then `journalctl -o cat | grep 'slot print_timing' | grep -o 'slot print_timing: id *[01].*draft acceptance.*'` to prove drafting is live on BOTH slots (`journalctl` default output truncates these lines; `-o cat` plus `grep -o` gets them whole).
- `grep -c 'ROCm error\|illegal memory\|ggml_abort'` on the current instance must be 0.

## Faults to watch, not to tune away

A cross-card boundary copy can fault (`amdgpu SDMA0 UTCL2 page fault` -> `ROCm error: an illegal memory access was encountered` -> `ggml_abort`) on a split that loads and answers `/health` perfectly. It is not an OOM — more headroom does not prevent it — so treat a split that has faulted as off-limits and re-measure rather than re-tuning. Also: a load that succeeds and then dies on the FIRST draft decode with an OOM is a sizing failure (the draft KV), not a broken drafter.