serverAppConfigs/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md

timestamp, type, title, description, tags

timestamp	type	title	description	tags
2026-09-17	reference	frank-medium — Batch and Ubatch Sweep	llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the deployed context shape, and the rebalance it forced. Prefill peaks at ubatch 512 and falls on BOTH sides.	benchmark llamacpp ubatch frank-medium dankserver layer-split

frank-medium — Batch and Ubatch Sweep

Run date: 2026-09-16. Harness: scripts/bench-batch-ubatch.sh (one llama-bench invocation per cell, a VRAM sampler per cell, the service stopped for the duration and restarted on exit). Tested at the deployed shape: 262144 ctx, two slots, --split-mode layer, f16 KV, flash attention on, DFlash2 loaded by the service but absent from the bench.

The question was how large the logical batch and the micro-batch may be, and what each buys. The answer is that --ubatch-size has an optimum on this stack, that the logical batch does not matter at all, and that the optimum is much cheaper than what was deployed — which then moved the layer split.

The shape of the measurement

A cell is a 64k prefill measured with 64k of KV already resident, plus a 128-token decode at that same depth:

llama-bench -m <UD-Q8_K_XL> -p 65536 -n 128 -d 65536 -b <b> -ub <ub> \
            -ngl 999 -dev ROCm0/ROCm1 -sm layer -ts 36/29 \
            -fa on -ctk f16 -ctv f16 -r 3 --no-warmup

-d 65536 is what puts depth in front of the measurement: a tg instance runs with n_prompt = 0, and an unfilled -d would measure a cold cache. The depth fill itself is excluded from the timing (and cached across reps), so the number is "prefill a 64k prompt into a 64k conversation", which is the agent-loop shape rather than a cold start.

Three llama-bench properties shaped the harness, all of them traps:

• No -c: the context is n_prompt + n_gen + n_depth, so a bench can never decide whether a configuration's buffers fit beside the deployed KV.
• A test that cannot allocate calls exit(1), taking the process with it — hence one invocation per cell.
• / separates devices within a configuration and , separates sweep values, so -ts 36,29 would have benched each card on its own.

Results — prefill, 64k depth (tok/s)

--batch-size	ub 256	ub 384	ub 512	ub 768	ub 1024	ub 1280	ub 1536	ub 1792	ub 2048
2048	—	—	1193.9	1147.9	1151.9	1141.7	1170.2	1128.8	1077.5
4096	—	—	1193.3	1129.6	1151.8	1143.7	1161.9	1151.9	1077.4
6144	1046.0	1068.0	1192.9	1135.0	1151.9	1151.4	1165.0	1147.7	1077.6
8192	—	—	1193.2	—	—	—	—	—	—

Per-cell standard deviations are 0.5–2.3 tok/s, i.e. under 0.2 %. Decode is 15.87–15.89 tok/s on every single cell, and the VRAM columns depend only on the ubatch (2048 / 4096 / 6144 at a fixed ubatch report identical free space).

Results — the same shape with no depth in front of it

N_DEPTH=0, same 64k prompt, two points for contrast:

--batch-size	ubatch	prefill (tok/s)	decode (tok/s)
6144	256	1543.5 ± 13.4	17.80
6144	1536	1722.7 ± 6.2	17.80

The ubatch curve has the same shape at both depths (below 512 is slower), and the two tables differ by a uniform ~32 %: that is what attending over 64k of resident KV costs. Depth is a tax on every configuration, not something the batch knobs can trade against.

What the data says

• --ubatch-size peaks at 512 and loses on both sides. Below it the SMs are not saturated (256 → 1046, 384 → 1068); at the top end 2048 is 10 % off the peak (1077). The middle is a shallow basin: 768–1792 sit at 1130–1170.
• The curve is deterministic, not noisy. 512 came out 1192.88 / 1193.23 / 1193.28 / 1193.88 across four different batch values, 1024 came out 1151.77 / 1151.91 / 1151.94, and 2048 came out 1077.38 / 1077.54 / 1077.59. Three independent batch columns agreeing to 0.4 tok/s is what makes the 2.4 % gap between 512 and 1536 real rather than run-to-run drift.
• The logical --batch-size is irrelevant here. At a fixed ubatch, 2048, 4096, 6144 and 8192 land within 2 % of each other — and use identical VRAM, because the compute buffers are sized by the ubatch alone.
• Decode does not move at all. 15.88 tok/s at 64k depth on every cell and 17.80 at depth 0, to ±0.05. Whatever this knob does, it does it to prefill.

What it costs at the deployed shape

Prefill rates do not decide a deployment; VRAM does. The same shape re-measured at 262144 over two slots (load, poll /health, read rocm-smi, kill — no requests):

Split	ubatch	GPU 0 free	GPU 1 free	gap
36,29 — deployed before	1536	4.71 GiB	4.91 GiB	0.20 GiB
36,29	512	6.01 GiB	7.32 GiB	1.31 GiB
35,30 — deployed now	512	6.44 GiB	6.90 GiB	0.46 GiB

Two things follow.

The ubatch move is free money. 1536 → 512 gives back 1.30 GiB on ROCm0 and 2.41 GiB on ROCm1 — 3.71 GiB in total — and the mean throughput goes up, not down. The asymmetry is the interesting part: ROCm1 pays nearly twice what ROCm0 does per unit of ubatch, because it holds the output layers and the drafter's graphs.

And it moves the balance, so the split had to follow. Those uneven give-backs left ROCm1 1.31 GiB lighter, and one layer is one -ts step: 36:29 became 35:30, which lands at 0.46 GiB — a single DeltaNet layer's worth of asymmetry, i.e. the floor on how well any split can balance this stack (0.43 GiB per DeltaNet layer, 1.43 per attention layer). The 0.20 GiB gap of the old configuration is not reachable at the new batch; 0.46 GiB is.

The decision

BATCH_SIZE=6144      # throughput-neutral and VRAM-neutral; left at the old value
UBATCH_SIZE=512      # the measured optimum: fastest AND cheapest
TENSOR_SPLIT=35,30   # rebalanced for the new ubatch, measured not interpolated

Verified live after the deploy: --tensor-split 35,30 --ubatch-size 512, two slots of 131072, 6.44 / 6.90 GiB free — matching the pre-deploy measurement to 0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.

The split: re-measure it at the deployed shape

The batch and the tensor split are coupled: compute buffers are not symmetric across the two cards, so any batch change moves the balance point. Order of work — sweep batch with llama-bench (above), THEN re-measure the split at the DEPLOYED shape (262144, two slots, drafter loaded) by loading the real service with a per-config env override and reading rocm-smi. A llama-bench number cannot decide the split: it has no --parallel, sizes its own context as n_prompt + n_gen + n_depth, and cannot load the drafter.

The settle gate is mandatory (above); wait for BOTH cards under ~1 GiB AND the port to be free (ss -ltn | grep :7770 until gone) before the next load, or the transient server dies with couldn't bind HTTP server socket. Free bytes follow card = weights_share + KV_share + constant + ubatch_term, where the ubatch term costs the drafter-carrying card roughly 1.85× the other (measured 2.41 GiB vs 1.30 GiB going 1536 → 512). Balance by measurement only: one -ts step is one layer, worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on an attention one, so the reachable minimum gap is about one DeltaNet layer, never zero. Reference: 36,29 @1536 → 4.71/4.91; 36,29 @512 → 6.01/7.32; 35,30 @512 → 6.44/6.90 (deployed).

Re-fit both ubatch coefficients in tests/frank_medium_unit.bats at the new deployed point and re-anchor the per-card constants; check neighbouring splits still FAIL the balance gate or the seam has stopped discriminating (keep the 1.5 GiB/card floor and the one-DeltaNet-layer slack). Deploy, then verify live: flags in /proc/<pid>/cmdline, /slots, VRAM matching the pre-deploy measurement, a tool call, two concurrent requests, draft acceptance lines on BOTH slots, 0 ROCm errors.

Per-card utilization (who is hot, and the split lever) — a different question from batch/prefill. Measure decode tok/s AND per-card rocm-smi --showuse at 2–3 neighbouring splits in ONE session. Pitfalls (each cost a re-run): run the transient unit as ROOT (a pkill -9 llama-server as dank auto-restarts the service and systemctl as dank fails "Access denied"); warm up 3 throwaway requests before the timed one (first 384-token decode after load measured 1.77 tok/s against a true ~32); rocm-smi --showmeminfo vram --csv columns are $2=Total, $3=Used — parsing the wrong column spins the settle gate the full 600 s; the util% is sampled post-hoc (which card finished last), so treat it as "which card is the pipeline tail", not a duty cycle.

What the split does and does not buy (measured 2026-09-17, 393216 / MTP n=3): the split is NOT a decode-throughput lever — 34:31, 35:30, 36:29 all sit in the same ~32 tok/s band. The x1 card runs hot (~91–97 %) at 35:30 because the MTP drafter + output layers + heavier compute buffers are pinned to the card holding the LAST layers — not because PCIe is the bottleneck (~5.2 MB of boundary activation crosses per 512-token ubatch). 36:29 makes the x16 card the utilized worker (42 %/3 %) at no throughput cost but spends headroom (0.45 GiB free, under the floor) — Frank's standing preference is that the card with the FASTEST PCIe lane should be the more utilized one; deploying it requires re-fitting the seam's slack/floor and re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.

Limitations

• llama-bench cannot run the drafter, the slots or the deployed context. Its decode figure is the drafterless baseline (15.9 at 64k depth against the service's ~44 with DFlash2 + ngram-mod), and it never exercises --parallel or the 16 GiB KV of a real 262144 deployment. The throughput comparison between batch values is sound; the absolute decode numbers are not the service's.
• Five cells of the intended grid were not run (batch 8192 above ubatch 512). Batch 8192 at ubatch 512 matches the other batch columns to 0.4 tok/s, and the sweep was stopped deliberately once the curve was established.
• A totals model cannot see contiguity. The earlier DSpark-era failure at 8192/2048 was a 2020 MiB pp buffer that found no room while every total looked fine. Nothing here rules that in or out; it only shows there is no reason to want a batch that large.
• A measurement like this needs a settle gate. Re-loading the model immediately after a kill fails with unable to allocate ROCm0 buffer — the previous instance's ~26 GiB is not returned instantly — which reads exactly like a configuration that does not fit. Wait for rocm-smi to show both cards near-empty before the next load.

See Also

• frank-medium-llamacpp-dflash2 — the deployment this tuned, and its split history
• frank-secondary-benchmark — the same sweep on a CUDA 3090, where a 16k prompt at depth 0 made prefill climb all the way to ubatch 2048. Depth is the difference: with no KV to attend over, per-pass overhead dominates and a bigger micro-batch wins; at 64k of resident KV it does not.
• Dual-R9700-Cooperation — why the cards are layer-split at all
• Dankserver System Specs — hardware inventory