The batch/ubatch sweep (scripts/bench-batch-ubatch.sh, 23 cells of a 28-cell
grid, 3 reps each) says the ubatch knob has an OPTIMUM, not a plateau:
ubatch 256 384 512 768 1024 1280 1536 1792 2048
tok/s 1046 1068 1193 1135 1152 1151 1165 1148 1078
The curve is remarkably reproducible -- 512 came out 1192.88 / 1193.23 /
1193.28 / 1193.88 across batch 2048, 4096, 6144 and 8192, and 2048 came out
1077.38 / 1077.54 / 1077.59 -- so this is a deterministic property of the
shape, not noise. The logical batch is throughput-neutral (2048..8192 all
within 2 % at a fixed ubatch) and VRAM-neutral (the compute buffers are sized
by ubatch alone). Decode is 15.88 tok/s at depth 65536 on every single cell.
So the deployed 6144x1536 was both 2.4 % slower than the optimum and 3.71 GiB
more expensive. Measured at the deployed shape (262144, 2 slots, DFlash2):
36,29 ubatch 1536 -> 4.71 / 4.91 GiB free (what was deployed)
36,29 ubatch 512 -> 6.01 / 7.32 GiB free (ubatch alone)
35,30 ubatch 512 -> 6.44 / 6.90 GiB free <- deployed
The ubatch move freed 1.30 GiB on ROCm0 but 2.41 GiB on ROCm1, because ROCm1
holds the output layers and the drafter's graphs -- so it shifted the balance
toward ROCm1 by 1.31 GiB and the split had to follow it one layer. 0.46 GiB is
one DeltaNet layer's worth of asymmetry, i.e. the granularity floor: no split
can be tuned finer than a layer, which is why 34,31 is not better.
Re-fitted the seam with both ubatch coefficients measured at this shape and
re-anchored it on the new deployed point (it reproduces 6.44 / 6.90 to
0.01 GiB). Every neighbouring split still fails it: 36,29 -> 1.85, 34,31 ->
0.94, 37,28 -> 3.24, 1,1 -> 3.03, 39,26 -> 6.03 GiB.
Also replaced the unit test's literal `TENSOR_SPLIT=36,29` assertion with the
shape it actually cares about -- two shares that add up to the model's 65
blocks, with ROCm1 taking the fewer -- so re-measuring the balance no longer
breaks a test about parallelism. The numeric contract lives in the seam test.
Frank
a11fae07d8