dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: ubatch 512, and rebalance the split to 35:30

Browse Source 
The batch/ubatch sweep (scripts/bench-batch-ubatch.sh, 23 cells of a 28-cell
grid, 3 reps each) says the ubatch knob has an OPTIMUM, not a plateau:

  ubatch   256    384    512    768   1024   1280   1536   1792   2048
  tok/s   1046   1068   1193   1135   1152   1151   1165   1148   1078

The curve is remarkably reproducible -- 512 came out 1192.88 / 1193.23 /
1193.28 / 1193.88 across batch 2048, 4096, 6144 and 8192, and 2048 came out
1077.38 / 1077.54 / 1077.59 -- so this is a deterministic property of the
shape, not noise. The logical batch is throughput-neutral (2048..8192 all
within 2 % at a fixed ubatch) and VRAM-neutral (the compute buffers are sized
by ubatch alone). Decode is 15.88 tok/s at depth 65536 on every single cell.

So the deployed 6144x1536 was both 2.4 % slower than the optimum and 3.71 GiB
more expensive. Measured at the deployed shape (262144, 2 slots, DFlash2):

  36,29 ubatch 1536 -> 4.71 / 4.91 GiB free   (what was deployed)
  36,29 ubatch 512  -> 6.01 / 7.32 GiB free   (ubatch alone)
  35,30 ubatch 512  -> 6.44 / 6.90 GiB free   <- deployed

The ubatch move freed 1.30 GiB on ROCm0 but 2.41 GiB on ROCm1, because ROCm1
holds the output layers and the drafter's graphs -- so it shifted the balance
toward ROCm1 by 1.31 GiB and the split had to follow it one layer. 0.46 GiB is
one DeltaNet layer's worth of asymmetry, i.e. the granularity floor: no split
can be tuned finer than a layer, which is why 34,31 is not better.

Re-fitted the seam with both ubatch coefficients measured at this shape and
re-anchored it on the new deployed point (it reproduces 6.44 / 6.90 to
0.01 GiB). Every neighbouring split still fails it: 36,29 -> 1.85, 34,31 ->
0.94, 37,28 -> 3.24, 1,1 -> 3.03, 39,26 -> 6.03 GiB.

Also replaced the unit test's literal `TENSOR_SPLIT=36,29` assertion with the
shape it actually cares about -- two shares that add up to the model's 65
blocks, with ROCm1 taking the fewer -- so re-measuring the balance no longer
breaks a test about parallelism. The numeric contract lives in the seam test.
This commit is contained in:
Frank
2026-09-17 00:21:48 -07:00
parent 2f012fba39
commit a11fae07d8
3 changed files with 142 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-medium.env
+64 -30
View File
@@ -57,15 +57,22 @@
# of what is seen below -- they are allocated at load, not per request, and they
# scale with --ubatch-size:
#
# CTX split batch slots drafter GPU 0 free GPU 1 free note
# CTX split batch slots drafter GPU 0 free GPU 1 free note (ubatch 1536)
# 262144 1,1 2048 1 DFlash2 8.64 GiB 4.43 GiB day's start
# 262144 39,26 6144 2 DFlash2 2.42 GiB 7.20 GiB
# 262144 38,27 6144 2 DFlash2 3.85 GiB 5.77 GiB
# 262144 37,28 6144 2 DFlash2 4.28 GiB 5.34 GiB
# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB <- deployed
# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB
# 262144 35,30 6144 2 DFlash2 5.14 GiB 4.48 GiB
# 262144 40,25 6144 2 DSpark 1.87 GiB 1.86 GiB (retired)
#
# ...and the same deployed shape re-measured at ubatch 512, after llama-bench
# showed 512 is the throughput optimum and costs 3.71 GiB less in total (see the
# batch block below):
#
# 262144 36,29 6144 512 2 DFlash2 6.01 GiB 7.32 GiB ubatch only
# 262144 35,30 6144 512 2 DFlash2 6.44 GiB 6.90 GiB <- deployed
#
# ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE. This is a hybrid
# stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a
# DeltaNet layer shifts ~0.43 GiB while an attention layer shifts ~1.43 GiB
@@ -79,30 +86,30 @@
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
#
# WHY THE SPLIT IS 36:29. Two effects both pull the balance BELOW an even
# split:
# WHY THE SPLIT IS 35:30. Three effects decide it, and all three are measured:
# * THE DRAFTER AND ITS OUTPUT LAYERS RIDE ROCm1. It carries the drafter's
# weights, the draft graph and the last target layers, so it must be given
# fewer.
# * THE BIGGER BATCH COSTS ROCm1 ABOUT TWICE WHAT IT COSTS ROCm0 (~3.3 vs
# ~1.7 GiB going 2048/512 -> 6144/1536 at 262144): ROCm1 holds the output
# layers AND the drafter's graphs. (Under the DSpark drafter that asymmetry
# dominated the picture, because DSpark also caches the WHOLE context; its
# KV alone was 5.0 GiB. DFlash2's window caps its KV at ~40 MiB, which
# freed ~4.3 GiB on ROCm1 and moved the balance four layers.)
# The layer-size granularity pushes back from the other side: 36:29 is the
# measured minimum (0.20 GiB gap, against 0.65 at 35:30 and 1.06 at 37:28).
# It also leaves the MOST ROOM -- this configuration clears the operator's
# 1.5 GiB-per-card floor by 3.2 GiB, so no headroom has to be bought out of
# CTX_SIZE and the full window stays.
# weights, the draft graph and the last target layers, so it takes fewer
# than half the layer count.
# * THE UBATCH BUFFERS COST ROCm1 MORE THAN ROCm0, and shrinking them pays
# ROCm1 back more: 1536 -> 512 freed 1.30 GiB on ROCm0 but 2.41 GiB on
# ROCm1 at this shape. Read the other way, that is exactly why the smaller
# batch moved the balance toward ROCm1 -- and cost ROCm0 one layer.
# * THE LAYER GRANULARITY IS THE FLOOR ON HOW WELL ANY SPLIT CAN BALANCE. One
# -ts step is one layer -- ~0.43 GiB DeltaNet, ~1.43 GiB attention -- so the
# reachable minimum gap is comparable to one step rather than to zero. 35:30
# sits at 0.46 GiB, i.e. one DeltaNet layer's worth, against 1.31 GiB at
# 36:29.
# It also leaves the most room: 6.44 / 6.90 GiB free clears the operator's 1.5
# GiB-per-card floor by ~5 GiB, so no headroom has to be bought out of CTX_SIZE
# and the full window stays.
#
# A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth
# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
# i.e. a fault in the cross-card boundary copy, this layer split's hot path. It
# has not reproduced (36:29 has served this file's verification battery, and the
# e2e gate), but the signature is worth recognising -- it is NOT an OOM, and no
# amount of headroom prevents it.
# has not reproduced (36:29 served this file's verification battery and the e2e
# gate, and 35:30 is the deployed shape), but the signature is worth recognising
# -- it is NOT an OOM, and no amount of headroom prevents it.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
@@ -129,12 +136,12 @@ HIP_VISIBLE_DEVICES=0,1
DEVICES=ROCm0,ROCm1
SPLIT_MODE=layer
# 36:29 -- the measured balance at the deployed batch, and the split that leaves
# 35:30 -- the measured balance at the deployed batch, and the split that leaves
# the most room: ROCm1 carries the drafter, its KV and the output layers, so it
# takes fewer target layers. See SIZING above for the measurement table; one
# -ts step is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so
# do not move it without re-measuring.
TENSOR_SPLIT=36,29
TENSOR_SPLIT=35,30
NGL=999
# Auto-fit has no business resizing a context we set explicitly.
@@ -148,15 +155,42 @@ FLASH_ATTN=on
CTX_SIZE=262144
PARALLEL=2
# 3x the build's defaults (2048/512). --ubatch-size is the physical batch the
# graph is built at and therefore what the compute buffers cost: at 262144 this
# setting needs ~1.7 GiB more on ROCm0 and ~3.3 GiB on ROCm1 than 2048/512 does
# -- which is what put the balance below an even split. (8192/2048 failed to
# allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had
# 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and with
# DFlash2's ~4.9 GiB of headroom it has NOT been re-measured.)
# MEASURED 2026-09-16 with scripts/bench-batch-ubatch.sh on this box: llama-bench,
# a 64k prefill plus a 128-token decode with 64k of KV already resident, 3 reps,
# one invocation per cell, the deployed split and f16 KV. --ubatch-size is the
# physical batch the graph is built at -- and the only one of the two knobs that
# costs VRAM:
#
# batch ubatch prefill tok/s GPU 0 / GPU 1 free
# 6144 1536 1164.96 +- 2.30 10.28 / 11.71 GiB
# 6144 768 1135.03 +- 1.09 11.29 / 13.28 GiB
# 6144 512 1192.93 +- 0.91 11.63 / 13.81 GiB <- deployed
# 6144 384 1068.04 +- 0.79 11.80 / 14.07 GiB
# 6144 256 1045.98 +- 1.69 11.97 / 14.33 GiB
# 4096 1536 1161.90 +- 0.96 10.28 / 11.71 GiB
# 2048 1536 1170.23 +- 0.86 10.28 / 11.71 GiB
# 2048 512 1193.88 +- 1.01 11.63 / 13.81 GiB
#
# THE PLATEAU STARTS AT 512. 512 -> 1536 buys ~2 % of prefill -- inside the
# run-to-run spread, and the 512 cells were the faster pair in BOTH batch
# columns -- while costing 1.35 GiB on ROCm0 and 2.10 GiB on ROCm1, so the
# deployment takes the cheap end. BELOW 512 the SMs are no longer saturated:
# 384 and 256 fall ~11 % and ~12 %. The logical batch is free either way --
# 2048, 4096 and 6144 land within 2 % of each other at a fixed ubatch and use
# the SAME VRAM, because the compute buffers are sized by ubatch alone.
#
# The prefill figure is a property of the SHAPE, not of this knob: the same
# sweep with no depth in front of it (depth 0) runs 1543 tok/s at ubatch 256 and
# 1723 at 1536 -- the plateau is in the same place, and the ~32 % between the two
# tables is what attending over 64k of resident KV costs. Decode is 15.88 tok/s
# at depth 65536 on EVERY ubatch (17.80 at depth 0), so this knob cannot touch
# generation at all.
#
# 8192/2048 stays untested under this drafter (the "2020 MiB pp buffer fails on
# ROCm1" note was measured under DSpark, when it had 0.08 GiB free), and there is
# no longer a reason to want it: batch buys nothing measurable here.
BATCH_SIZE=6144
UBATCH_SIZE=1536
UBATCH_SIZE=512
# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
NUM_THREADS=8