serverAppConfigs/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md

---
timestamp: 2026-09-17
type: reference
title: frank-medium — Batch and Ubatch Sweep
description: llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the deployed context shape, and the rebalance it forced. Prefill peaks at ubatch 512 and falls on BOTH sides.
tags:
  - benchmark
  - llamacpp
  - ubatch
  - frank-medium
  - dankserver
  - layer-split
---

# frank-medium — Batch and Ubatch Sweep

Run date: 2026-09-16. Harness: `scripts/bench-batch-ubatch.sh` (one
`llama-bench` invocation per cell, a VRAM sampler per cell, the service stopped
for the duration and restarted on exit). Tested at the deployed shape: 262144
ctx, two slots, `--split-mode layer`, f16 KV, flash attention on, DFlash2
loaded by the service but absent from the bench.

The question was how large the logical batch and the micro-batch may be, and
what each buys. The answer is that `--ubatch-size` has an **optimum** on this
stack, that the logical batch does not matter at all, and that the optimum is
much cheaper than what was deployed — which then moved the layer split.

## The shape of the measurement

A cell is a **64k prefill measured with 64k of KV already resident**, plus a
128-token decode at that same depth:

```
llama-bench -m <UD-Q8_K_XL> -p 65536 -n 128 -d 65536 -b <b> -ub <ub> \
            -ngl 999 -dev ROCm0/ROCm1 -sm layer -ts 36/29 \
            -fa on -ctk f16 -ctv f16 -r 3 --no-warmup
```

`-d 65536` is what puts depth in front of the measurement: a `tg` instance runs
with `n_prompt = 0`, and an unfilled `-d` would measure a cold cache. The depth
fill itself is excluded from the timing (and cached across reps), so the number
is "prefill a 64k prompt into a 64k conversation", which is the agent-loop shape
rather than a cold start.

Three `llama-bench` properties shaped the harness, all of them traps:

- **No `-c`**: the context is `n_prompt + n_gen + n_depth`, so a bench can never
  decide whether a configuration's buffers fit beside the *deployed* KV.
- **A test that cannot allocate calls `exit(1)`**, taking the process with it —
  hence one invocation per cell.
- **`/` separates devices within a configuration and `,` separates sweep
  values**, so `-ts 36,29` would have benched each card on its own.

## Results — prefill, 64k depth (tok/s)

| `--batch-size` | ub 256 | ub 384 | **ub 512** | ub 768 | ub 1024 | ub 1280 | ub 1536 | ub 1792 | ub 2048 |
|---|---|---|---|---|---|---|---|---|---|
| 2048 | — | — | **1193.9** | 1147.9 | 1151.9 | 1141.7 | 1170.2 | 1128.8 | 1077.5 |
| 4096 | — | — | **1193.3** | 1129.6 | 1151.8 | 1143.7 | 1161.9 | 1151.9 | 1077.4 |
| 6144 | 1046.0 | 1068.0 | **1192.9** | 1135.0 | 1151.9 | 1151.4 | 1165.0 | 1147.7 | 1077.6 |
| 8192 | — | — | **1193.2** | — | — | — | — | — | — |

Per-cell standard deviations are 0.5–2.3 tok/s, i.e. under 0.2 %. **Decode is
15.87–15.89 tok/s on every single cell**, and the VRAM columns depend only on
the ubatch (2048 / 4096 / 6144 at a fixed ubatch report identical free space).

## Results — the same shape with no depth in front of it

`N_DEPTH=0`, same 64k prompt, two points for contrast:

| `--batch-size` | ubatch | prefill (tok/s) | decode (tok/s) |
|---|---|---|---|
| 6144 | 256 | 1543.5 ± 13.4 | 17.80 |
| 6144 | 1536 | 1722.7 ± 6.2 | 17.80 |

The **ubatch curve has the same shape at both depths** (below 512 is slower),
and the two tables differ by a uniform ~32 %: that is what attending over 64k of
resident KV costs. Depth is a tax on every configuration, not something the
batch knobs can trade against.

## What the data says

- **`--ubatch-size` peaks at 512 and loses on both sides.** Below it the SMs are
  not saturated (256 → 1046, 384 → 1068); at the top end 2048 is 10 % off the
  peak (1077). The middle is a shallow basin: 768–1792 sit at 1130–1170.
- **The curve is deterministic, not noisy.** 512 came out 1192.88 / 1193.23 /
  1193.28 / 1193.88 across four different batch values, 1024 came out 1151.77 /
  1151.91 / 1151.94, and 2048 came out 1077.38 / 1077.54 / 1077.59. Three
  independent batch columns agreeing to 0.4 tok/s is what makes the 2.4 % gap
  between 512 and 1536 real rather than run-to-run drift.
- **The logical `--batch-size` is irrelevant here.** At a fixed ubatch, 2048,
  4096, 6144 and 8192 land within 2 % of each other — and use *identical* VRAM,
  because the compute buffers are sized by the ubatch alone.
- **Decode does not move at all.** 15.88 tok/s at 64k depth on every cell and
  17.80 at depth 0, to ±0.05. Whatever this knob does, it does it to prefill.

## What it costs at the deployed shape

Prefill rates do not decide a deployment; VRAM does. The same shape
re-measured at 262144 over two slots (load, poll `/health`, read `rocm-smi`,
kill — no requests):

| Split | ubatch | GPU 0 free | GPU 1 free | gap |
|---|---|---|---|---|
| 36,29 — deployed before | 1536 | 4.71 GiB | 4.91 GiB | 0.20 GiB |
| 36,29 | **512** | 6.01 GiB | 7.32 GiB | 1.31 GiB |
| **35,30 — deployed now** | **512** | **6.44 GiB** | **6.90 GiB** | **0.46 GiB** |

Two things follow.

**The ubatch move is free money.** 1536 → 512 gives back 1.30 GiB on ROCm0 and
2.41 GiB on ROCm1 — 3.71 GiB in total — and the mean throughput goes *up*, not
down. The asymmetry is the interesting part: ROCm1 pays nearly twice what ROCm0
does per unit of ubatch, because it holds the output layers and the drafter's
graphs.

**And it moves the balance, so the split had to follow.** Those uneven
give-backs left ROCm1 1.31 GiB lighter, and one layer is one `-ts` step: 36:29
became 35:30, which lands at 0.46 GiB — a *single DeltaNet layer's* worth of
asymmetry, i.e. the floor on how well any split can balance this stack (0.43 GiB
per DeltaNet layer, 1.43 per attention layer). The 0.20 GiB gap of the old
configuration is not reachable at the new batch; 0.46 GiB is.

## The decision

```ini
BATCH_SIZE=6144      # throughput-neutral and VRAM-neutral; left at the old value
UBATCH_SIZE=512      # the measured optimum: fastest AND cheapest
TENSOR_SPLIT=35,30   # rebalanced for the new ubatch, measured not interpolated
```

Verified live after the deploy: `--tensor-split 35,30 --ubatch-size 512`, two
slots of 131072, `6.44 / 6.90 GiB` free — matching the pre-deploy measurement to
0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on
both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.

## The split: re-measure it at the deployed shape

The batch and the tensor split are **coupled**: compute buffers are not
symmetric across the two cards, so any batch change moves the balance point.
Order of work — sweep batch with llama-bench (above), THEN re-measure the
split at the DEPLOYED shape (262144, two slots, drafter loaded) by loading
the real service with a per-config env override and reading `rocm-smi`. A
llama-bench number cannot decide the split: it has no `--parallel`, sizes
its own context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.

The settle gate is mandatory (above); wait for BOTH cards under ~1 GiB AND
the port to be free (`ss -ltn | grep :7770` until gone) before the next
load, or the transient server dies with `couldn't bind HTTP server socket`.
Free bytes follow `card = weights_share + KV_share + constant + ubatch_term`,
where the ubatch term costs the drafter-carrying card roughly **1.85×** the
other (measured 2.41 GiB vs 1.30 GiB going 1536 → 512). Balance by
measurement only: one `-ts` step is one layer, worth ~0.43 GiB on a DeltaNet
boundary and ~1.43 GiB on an attention one, so the reachable minimum gap is
about one DeltaNet layer, never zero. Reference: 36,29 @1536 → 4.71/4.91;
36,29 @512 → 6.01/7.32; **35,30 @512 → 6.44/6.90 (deployed)**.

Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` at the new
deployed point and re-anchor the per-card constants; check neighbouring
splits still FAIL the balance gate or the seam has stopped discriminating
(keep the 1.5 GiB/card floor and the one-DeltaNet-layer slack). Deploy,
then verify live: flags in `/proc/<pid>/cmdline`, `/slots`, VRAM matching
the pre-deploy measurement, a tool call, two concurrent requests,
`draft acceptance` lines on BOTH slots, 0 ROCm errors.

**Per-card utilization (who is hot, and the split lever)** — a different
question from batch/prefill. Measure decode tok/s AND per-card
`rocm-smi --showuse` at 2–3 neighbouring splits in ONE session. Pitfalls
(each cost a re-run): run the transient unit as ROOT (a `pkill -9
llama-server` as dank auto-restarts the service and `systemctl` as dank
fails "Access denied"); warm up 3 throwaway requests before the timed one
(first 384-token decode after load measured 1.77 tok/s against a true ~32);
`rocm-smi --showmeminfo vram --csv` columns are `$2`=Total, `$3`=**Used** —
parsing the wrong column spins the settle gate the full 600 s; the util% is
sampled post-hoc (which card finished last), so treat it as "which card is
the pipeline tail", not a duty cycle.

**What the split does and does not buy (measured 2026-09-17, 393216 /
MTP n=3):** the split is NOT a decode-throughput lever — 34:31, 35:30, 36:29
all sit in the same ~32 tok/s band. The x1 card runs hot (~91–97 %) at
35:30 because the MTP drafter + output layers + heavier compute buffers are
pinned to the card holding the LAST layers — not because PCIe is the
bottleneck (~5.2 MB of boundary activation crosses per 512-token ubatch).
36:29 makes the x16 card the utilized worker (42 %/3 %) at no throughput
cost but spends headroom (0.45 GiB free, under the floor) — Frank's standing
preference is that the card with the FASTEST PCIe lane should be the more
utilized one; deploying it requires re-fitting the seam's slack/floor and
re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.

## Limitations

- **`llama-bench` cannot run the drafter, the slots or the deployed context.**
  Its decode figure is the *drafterless* baseline (15.9 at 64k depth against the
  service's ~44 with DFlash2 + ngram-mod), and it never exercises `--parallel` or
  the 16 GiB KV of a real 262144 deployment. The throughput comparison between
  batch values is sound; the absolute decode numbers are not the service's.
- **Five cells of the intended grid were not run** (batch 8192 above ubatch 512).
  Batch 8192 at ubatch 512 matches the other batch columns to 0.4 tok/s, and the
  sweep was stopped deliberately once the curve was established.
- **A totals model cannot see contiguity.** The earlier DSpark-era failure at
  8192/2048 was a 2020 MiB pp buffer that found no room while every total looked
  fine. Nothing here rules that in or out; it only shows there is no reason to
  want a batch that large.
- **A measurement like this needs a settle gate.** Re-loading the model
  immediately after a kill fails with `unable to allocate ROCm0 buffer` — the
  previous instance's ~26 GiB is not returned instantly — which reads exactly
  like a configuration that does not fit. Wait for `rocm-smi` to show both cards
  near-empty before the next load.

## See Also

- [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split
  history
- [[frank-zealot]] — the same sweep on a CUDA 3090, where a 16k prompt
  at depth 0 made prefill climb all the way to ubatch 2048. Depth is the
  difference: with no KV to attend over, per-pass overhead dominates and a
  bigger micro-batch wins; at 64k of resident KV it does not.
- [[Dual-R9700-Cooperation]] — why the cards are layer-split at all
- [[Dankserver System Specs]] — hardware inventory