serverAppConfigs/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md

timestamp, type, title, description, tags

timestamp	type	title	description	tags
2026-09-17	reference	frank-medium — Batch and Ubatch Sweep	llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the deployed context shape, and the rebalance it forced. Prefill peaks at ubatch 512 and falls on BOTH sides.	benchmark llamacpp ubatch frank-medium dankserver layer-split

frank-medium — Batch and Ubatch Sweep

Run date: 2026-09-16. Harness: scripts/bench-batch-ubatch.sh (one llama-bench invocation per cell, a VRAM sampler per cell, the service stopped for the duration and restarted on exit). Tested at the deployed shape: 262144 ctx, two slots, --split-mode layer, f16 KV, flash attention on, DFlash2 loaded by the service but absent from the bench.

The question was how large the logical batch and the micro-batch may be, and what each buys. The answer is that --ubatch-size has an optimum on this stack, that the logical batch does not matter at all, and that the optimum is much cheaper than what was deployed — which then moved the layer split.

The shape of the measurement

A cell is a 64k prefill measured with 64k of KV already resident, plus a 128-token decode at that same depth:

llama-bench -m <UD-Q8_K_XL> -p 65536 -n 128 -d 65536 -b <b> -ub <ub> \
            -ngl 999 -dev ROCm0/ROCm1 -sm layer -ts 36/29 \
            -fa on -ctk f16 -ctv f16 -r 3 --no-warmup

-d 65536 is what puts depth in front of the measurement: a tg instance runs with n_prompt = 0, and an unfilled -d would measure a cold cache. The depth fill itself is excluded from the timing (and cached across reps), so the number is "prefill a 64k prompt into a 64k conversation", which is the agent-loop shape rather than a cold start.

Three llama-bench properties shaped the harness, all of them traps:

• No -c: the context is n_prompt + n_gen + n_depth, so a bench can never decide whether a configuration's buffers fit beside the deployed KV.
• A test that cannot allocate calls exit(1), taking the process with it — hence one invocation per cell.
• / separates devices within a configuration and , separates sweep values, so -ts 36,29 would have benched each card on its own.

Results — prefill, 64k depth (tok/s)

--batch-size	ub 256	ub 384	ub 512	ub 768	ub 1024	ub 1280	ub 1536	ub 1792	ub 2048
2048	—	—	1193.9	1147.9	1151.9	1141.7	1170.2	1128.8	1077.5
4096	—	—	1193.3	1129.6	1151.8	1143.7	1161.9	1151.9	1077.4
6144	1046.0	1068.0	1192.9	1135.0	1151.9	1151.4	1165.0	1147.7	1077.6
8192	—	—	1193.2	—	—	—	—	—	—

Per-cell standard deviations are 0.5–2.3 tok/s, i.e. under 0.2 %. Decode is 15.87–15.89 tok/s on every single cell, and the VRAM columns depend only on the ubatch (2048 / 4096 / 6144 at a fixed ubatch report identical free space).

Results — the same shape with no depth in front of it

N_DEPTH=0, same 64k prompt, two points for contrast:

--batch-size	ubatch	prefill (tok/s)	decode (tok/s)
6144	256	1543.5 ± 13.4	17.80
6144	1536	1722.7 ± 6.2	17.80

The ubatch curve has the same shape at both depths (below 512 is slower), and the two tables differ by a uniform ~32 %: that is what attending over 64k of resident KV costs. Depth is a tax on every configuration, not something the batch knobs can trade against.

What the data says

• --ubatch-size peaks at 512 and loses on both sides. Below it the SMs are not saturated (256 → 1046, 384 → 1068); at the top end 2048 is 10 % off the peak (1077). The middle is a shallow basin: 768–1792 sit at 1130–1170.
• The curve is deterministic, not noisy. 512 came out 1192.88 / 1193.23 / 1193.28 / 1193.88 across four different batch values, 1024 came out 1151.77 / 1151.91 / 1151.94, and 2048 came out 1077.38 / 1077.54 / 1077.59. Three independent batch columns agreeing to 0.4 tok/s is what makes the 2.4 % gap between 512 and 1536 real rather than run-to-run drift.
• The logical --batch-size is irrelevant here. At a fixed ubatch, 2048, 4096, 6144 and 8192 land within 2 % of each other — and use identical VRAM, because the compute buffers are sized by the ubatch alone.
• Decode does not move at all. 15.88 tok/s at 64k depth on every cell and 17.80 at depth 0, to ±0.05. Whatever this knob does, it does it to prefill.

What it costs at the deployed shape

Prefill rates do not decide a deployment; VRAM does. The same shape re-measured at 262144 over two slots (load, poll /health, read rocm-smi, kill — no requests):

Split	ubatch	GPU 0 free	GPU 1 free	gap
36,29 — deployed before	1536	4.71 GiB	4.91 GiB	0.20 GiB
36,29	512	6.01 GiB	7.32 GiB	1.31 GiB
35,30 — deployed now	512	6.44 GiB	6.90 GiB	0.46 GiB

Two things follow.

The ubatch move is free money. 1536 → 512 gives back 1.30 GiB on ROCm0 and 2.41 GiB on ROCm1 — 3.71 GiB in total — and the mean throughput goes up, not down. The asymmetry is the interesting part: ROCm1 pays nearly twice what ROCm0 does per unit of ubatch, because it holds the output layers and the drafter's graphs.

And it moves the balance, so the split had to follow. Those uneven give-backs left ROCm1 1.31 GiB lighter, and one layer is one -ts step: 36:29 became 35:30, which lands at 0.46 GiB — a single DeltaNet layer's worth of asymmetry, i.e. the floor on how well any split can balance this stack (0.43 GiB per DeltaNet layer, 1.43 per attention layer). The 0.20 GiB gap of the old configuration is not reachable at the new batch; 0.46 GiB is.

The decision

BATCH_SIZE=6144      # throughput-neutral and VRAM-neutral; left at the old value
UBATCH_SIZE=512      # the measured optimum: fastest AND cheapest
TENSOR_SPLIT=35,30   # rebalanced for the new ubatch, measured not interpolated

Verified live after the deploy: --tensor-split 35,30 --ubatch-size 512, two slots of 131072, 6.44 / 6.90 GiB free — matching the pre-deploy measurement to 0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.

Limitations

• llama-bench cannot run the drafter, the slots or the deployed context. Its decode figure is the drafterless baseline (15.9 at 64k depth against the service's ~44 with DFlash2 + ngram-mod), and it never exercises --parallel or the 16 GiB KV of a real 262144 deployment. The throughput comparison between batch values is sound; the absolute decode numbers are not the service's.
• Five cells of the intended grid were not run (batch 8192 above ubatch 512). Batch 8192 at ubatch 512 matches the other batch columns to 0.4 tok/s, and the sweep was stopped deliberately once the curve was established.
• A totals model cannot see contiguity. The earlier DSpark-era failure at 8192/2048 was a 2020 MiB pp buffer that found no room while every total looked fine. Nothing here rules that in or out; it only shows there is no reason to want a batch that large.
• A measurement like this needs a settle gate. Re-loading the model immediately after a kill fails with unable to allocate ROCm0 buffer — the previous instance's ~26 GiB is not returned instantly — which reads exactly like a configuration that does not fit. Wait for rocm-smi to show both cards near-empty before the next load.

See Also

• frank-medium-llamacpp-dflash2 — the deployment this tuned, and its split history
• frank-small-benchmark — the same sweep on a CUDA 3090, where a 16k prompt at depth 0 made prefill climb all the way to ubatch 2048. Depth is the difference: with no KV to attend over, per-pass overhead dominates and a bigger micro-batch wins; at 64k of resident KV it does not.
• Dual-R9700-Cooperation — why the cards are layer-split at all
• Dankserver System Specs — hardware inventory