dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: ubatch 512, and rebalance the split to 35:30

Browse Source 
The batch/ubatch sweep (scripts/bench-batch-ubatch.sh, 23 cells of a 28-cell
grid, 3 reps each) says the ubatch knob has an OPTIMUM, not a plateau:

  ubatch   256    384    512    768   1024   1280   1536   1792   2048
  tok/s   1046   1068   1193   1135   1152   1151   1165   1148   1078

The curve is remarkably reproducible -- 512 came out 1192.88 / 1193.23 /
1193.28 / 1193.88 across batch 2048, 4096, 6144 and 8192, and 2048 came out
1077.38 / 1077.54 / 1077.59 -- so this is a deterministic property of the
shape, not noise. The logical batch is throughput-neutral (2048..8192 all
within 2 % at a fixed ubatch) and VRAM-neutral (the compute buffers are sized
by ubatch alone). Decode is 15.88 tok/s at depth 65536 on every single cell.

So the deployed 6144x1536 was both 2.4 % slower than the optimum and 3.71 GiB
more expensive. Measured at the deployed shape (262144, 2 slots, DFlash2):

  36,29 ubatch 1536 -> 4.71 / 4.91 GiB free   (what was deployed)
  36,29 ubatch 512  -> 6.01 / 7.32 GiB free   (ubatch alone)
  35,30 ubatch 512  -> 6.44 / 6.90 GiB free   <- deployed

The ubatch move freed 1.30 GiB on ROCm0 but 2.41 GiB on ROCm1, because ROCm1
holds the output layers and the drafter's graphs -- so it shifted the balance
toward ROCm1 by 1.31 GiB and the split had to follow it one layer. 0.46 GiB is
one DeltaNet layer's worth of asymmetry, i.e. the granularity floor: no split
can be tuned finer than a layer, which is why 34,31 is not better.

Re-fitted the seam with both ubatch coefficients measured at this shape and
re-anchored it on the new deployed point (it reproduces 6.44 / 6.90 to
0.01 GiB). Every neighbouring split still fails it: 36,29 -> 1.85, 34,31 ->
0.94, 37,28 -> 3.24, 1,1 -> 3.03, 39,26 -> 6.03 GiB.

Also replaced the unit test's literal `TENSOR_SPLIT=36,29` assertion with the
shape it actually cares about -- two shares that add up to the model's 65
blocks, with ROCm1 taking the fewer -- so re-measuring the balance no longer
breaks a test about parallelism. The numeric contract lives in the seam test.
This commit is contained in:
Frank
2026-09-17 00:21:48 -07:00
parent 2f012fba39
commit a11fae07d8
3 changed files with 142 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-medium.env
+64 -30
View File
@@ -57,15 +57,22 @@
# of what is seen below -- they are allocated at load, not per request, and they
# scale with --ubatch-size:
#
# CTX split batch slots drafter GPU 0 free GPU 1 free note
# CTX split batch slots drafter GPU 0 free GPU 1 free note (ubatch 1536)
# 262144 1,1 2048 1 DFlash2 8.64 GiB 4.43 GiB day's start
# 262144 39,26 6144 2 DFlash2 2.42 GiB 7.20 GiB
# 262144 38,27 6144 2 DFlash2 3.85 GiB 5.77 GiB
# 262144 37,28 6144 2 DFlash2 4.28 GiB 5.34 GiB
# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB <- deployed
# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB
# 262144 35,30 6144 2 DFlash2 5.14 GiB 4.48 GiB
# 262144 40,25 6144 2 DSpark 1.87 GiB 1.86 GiB (retired)
#
# ...and the same deployed shape re-measured at ubatch 512, after llama-bench
# showed 512 is the throughput optimum and costs 3.71 GiB less in total (see the
# batch block below):
#
# 262144 36,29 6144 512 2 DFlash2 6.01 GiB 7.32 GiB ubatch only
# 262144 35,30 6144 512 2 DFlash2 6.44 GiB 6.90 GiB <- deployed
#
# ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE. This is a hybrid
# stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a
# DeltaNet layer shifts ~0.43 GiB while an attention layer shifts ~1.43 GiB
@@ -79,30 +86,30 @@
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
#
# WHY THE SPLIT IS 36:29. Two effects both pull the balance BELOW an even
# split:
# WHY THE SPLIT IS 35:30. Three effects decide it, and all three are measured:
# * THE DRAFTER AND ITS OUTPUT LAYERS RIDE ROCm1. It carries the drafter's
# weights, the draft graph and the last target layers, so it must be given
# fewer.
# * THE BIGGER BATCH COSTS ROCm1 ABOUT TWICE WHAT IT COSTS ROCm0 (~3.3 vs
# ~1.7 GiB going 2048/512 -> 6144/1536 at 262144): ROCm1 holds the output
# layers AND the drafter's graphs. (Under the DSpark drafter that asymmetry
# dominated the picture, because DSpark also caches the WHOLE context; its
# KV alone was 5.0 GiB. DFlash2's window caps its KV at ~40 MiB, which
# freed ~4.3 GiB on ROCm1 and moved the balance four layers.)
# The layer-size granularity pushes back from the other side: 36:29 is the
# measured minimum (0.20 GiB gap, against 0.65 at 35:30 and 1.06 at 37:28).
# It also leaves the MOST ROOM -- this configuration clears the operator's
# 1.5 GiB-per-card floor by 3.2 GiB, so no headroom has to be bought out of
# CTX_SIZE and the full window stays.
# weights, the draft graph and the last target layers, so it takes fewer
# than half the layer count.
# * THE UBATCH BUFFERS COST ROCm1 MORE THAN ROCm0, and shrinking them pays
# ROCm1 back more: 1536 -> 512 freed 1.30 GiB on ROCm0 but 2.41 GiB on
# ROCm1 at this shape. Read the other way, that is exactly why the smaller
# batch moved the balance toward ROCm1 -- and cost ROCm0 one layer.
# * THE LAYER GRANULARITY IS THE FLOOR ON HOW WELL ANY SPLIT CAN BALANCE. One
# -ts step is one layer -- ~0.43 GiB DeltaNet, ~1.43 GiB attention -- so the
# reachable minimum gap is comparable to one step rather than to zero. 35:30
# sits at 0.46 GiB, i.e. one DeltaNet layer's worth, against 1.31 GiB at
# 36:29.
# It also leaves the most room: 6.44 / 6.90 GiB free clears the operator's 1.5
# GiB-per-card floor by ~5 GiB, so no headroom has to be bought out of CTX_SIZE
# and the full window stays.
#
# A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth
# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
# i.e. a fault in the cross-card boundary copy, this layer split's hot path. It
# has not reproduced (36:29 has served this file's verification battery, and the
# e2e gate), but the signature is worth recognising -- it is NOT an OOM, and no
# amount of headroom prevents it.
# has not reproduced (36:29 served this file's verification battery and the e2e
# gate, and 35:30 is the deployed shape), but the signature is worth recognising
# -- it is NOT an OOM, and no amount of headroom prevents it.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
@@ -129,12 +136,12 @@ HIP_VISIBLE_DEVICES=0,1
DEVICES=ROCm0,ROCm1
SPLIT_MODE=layer
# 36:29 -- the measured balance at the deployed batch, and the split that leaves
# 35:30 -- the measured balance at the deployed batch, and the split that leaves
# the most room: ROCm1 carries the drafter, its KV and the output layers, so it
# takes fewer target layers. See SIZING above for the measurement table; one
# -ts step is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so
# do not move it without re-measuring.
TENSOR_SPLIT=36,29
TENSOR_SPLIT=35,30
NGL=999
# Auto-fit has no business resizing a context we set explicitly.
@@ -148,15 +155,42 @@ FLASH_ATTN=on
CTX_SIZE=262144
PARALLEL=2
# 3x the build's defaults (2048/512). --ubatch-size is the physical batch the
# graph is built at and therefore what the compute buffers cost: at 262144 this
# setting needs ~1.7 GiB more on ROCm0 and ~3.3 GiB on ROCm1 than 2048/512 does
# -- which is what put the balance below an even split. (8192/2048 failed to
# allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had
# 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and with
# DFlash2's ~4.9 GiB of headroom it has NOT been re-measured.)
# MEASURED 2026-09-16 with scripts/bench-batch-ubatch.sh on this box: llama-bench,
# a 64k prefill plus a 128-token decode with 64k of KV already resident, 3 reps,
# one invocation per cell, the deployed split and f16 KV. --ubatch-size is the
# physical batch the graph is built at -- and the only one of the two knobs that
# costs VRAM:
#
# batch ubatch prefill tok/s GPU 0 / GPU 1 free
# 6144 1536 1164.96 +- 2.30 10.28 / 11.71 GiB
# 6144 768 1135.03 +- 1.09 11.29 / 13.28 GiB
# 6144 512 1192.93 +- 0.91 11.63 / 13.81 GiB <- deployed
# 6144 384 1068.04 +- 0.79 11.80 / 14.07 GiB
# 6144 256 1045.98 +- 1.69 11.97 / 14.33 GiB
# 4096 1536 1161.90 +- 0.96 10.28 / 11.71 GiB
# 2048 1536 1170.23 +- 0.86 10.28 / 11.71 GiB
# 2048 512 1193.88 +- 1.01 11.63 / 13.81 GiB
#
# THE PLATEAU STARTS AT 512. 512 -> 1536 buys ~2 % of prefill -- inside the
# run-to-run spread, and the 512 cells were the faster pair in BOTH batch
# columns -- while costing 1.35 GiB on ROCm0 and 2.10 GiB on ROCm1, so the
# deployment takes the cheap end. BELOW 512 the SMs are no longer saturated:
# 384 and 256 fall ~11 % and ~12 %. The logical batch is free either way --
# 2048, 4096 and 6144 land within 2 % of each other at a fixed ubatch and use
# the SAME VRAM, because the compute buffers are sized by ubatch alone.
#
# The prefill figure is a property of the SHAPE, not of this knob: the same
# sweep with no depth in front of it (depth 0) runs 1543 tok/s at ubatch 256 and
# 1723 at 1536 -- the plateau is in the same place, and the ~32 % between the two
# tables is what attending over 64k of resident KV costs. Decode is 15.88 tok/s
# at depth 65536 on EVERY ubatch (17.80 at depth 0), so this knob cannot touch
# generation at all.
#
# 8192/2048 stays untested under this drafter (the "2020 MiB pp buffer fails on
# ROCm1" note was measured under DSpark, when it had 0.08 GiB free), and there is
# no longer a reason to want it: batch buys nothing measurable here.
BATCH_SIZE=6144
UBATCH_SIZE=1536
UBATCH_SIZE=512
# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
NUM_THREADS=8
serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md
+41
View File
@@ -31,6 +31,16 @@ P2P operation can be dispatched on card #2. Two paths restore cooperation:
2. **A board whose CPU x16 bifurcates to x8/x8** (X670E/X870E class,
~$250-400). Best performance. This board has no x8/x8 mode.
The M.2_2 half of path 1 rests on a **vendor claim, not on this host**: ASUS
lists M.2_1 and M.2_2 as CPU-connected (PCIe 5.0 x4 and 4.0 x4) and M.2_3 as
chipset, while an empty slot does not enumerate — so nothing on a running
system can confirm it until a card is in it. And CPU-attached does not imply
`Routing+`: `00:08.3` is CPU-side and reads `Routing-`. The precedent worth
trusting is `00:01.2`, the CPU M.2_1 port, which is `Routing+`. Note too what
the riser buys: **tensor parallelism** (~40 % decode), not "both cards on one
model" — the layer split already delivers that, so this is a performance
project, not a capability one.
## The gate, in source
```cpp
@@ -59,6 +69,37 @@ CPU must advertise `AtomicOpsCap: Routing+`. On this board card #2's path
includes `05:00.0` and `06:09.0`, both `Routing-` — a **read-only capability
bit**, so no software can enable it.
### The same gate, as this board reports it (2026-09-16)
`lspci -vv` on dankserver. Card 0's chain is root port `00:01.1` → `01:00.0` →
`02:00.0` → `03:00.0`; card 1's is `00:02.1` → `05:00.0` → `06:00.0` →
`06:09.0` → `09:00.0` → `0a:00.0` → `0b:00.0`:
| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
|---|---|---|---|---|
| `00:01.1` | CPU root port → PCIEX16_1 | Routing+ | — | x16 |
| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | x16 |
| `00:02.1` | CPU root port → chipset | **Routing+** | — | x4 |
| `05:00.0`, `06:00.0`, `06:09.0` | B650 bridges | **Routing−** | — | x1 at `06:09.0` |
| `09:00.0`, `0a:00.0` | PCIe switch ports | Routing+ | — | x1 (downgraded) |
| `0b:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn−** | x16 |
Three things fall out of that table, none of them in the source alone:
- **The CPU side is fine** — `00:02.1` is `Routing+`. The wall is inside the
chipset, not the platform, which is why a slot on CPU lanes is the fix.
- **The endpoint's own bit differs between the cards, and that is the kernel's
doing.** `pci_enable_atomic_ops_to_root()` walks the bridge chain and refuses
to set Requester Enable when any bridge cannot route atomics. ROCclr's
`pcie_atomics_` and the endpoint's `ReqEn` are two faces of one cause, and it
makes `lspci -vv -s <bdf> | grep AtomicOpsCtl` a one-command diagnosis: on
this host card 0 reads `ReqEn+` and card 1 `ReqEn−`.
- **Card 1's effective link is x1**, not the x16 its own capability block
advertises: `06:09.0` trained at x1, so ~2 GB/s is the real ceiling. That is a
bandwidth fact rather than an atomics one, and it only matters for
collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at
x4. The layer split's boundary copy does not care.
## Measured matrix
Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no
tests/frank_medium_unit.bats
+37 -16
View File
@@ -47,7 +47,18 @@ DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter
@test "both R9700s serve ONE model, layer-split inside one process" {
grep -q '^SPLIT_MODE=layer$' "$ENV_FILE"
grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE"
grep -q '^TENSOR_SPLIT=36,29$' "$ENV_FILE"
# The split is a layer PAIR: the shares add up to the model's 65 blocks (one
# -ts unit per layer) and ROCm1 -- the card that also carries the drafter and
# the output layers -- takes the FEWER of the two. WHICH pair balances the
# cards is the seam test's job, which recomputes it from this file; pinning a
# literal here would only mean that re-measuring the balance breaks a test
# about parallelism.
local split_hi split_lo
split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")"
split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")"
[ -n "$split_hi" ] && [ -n "$split_lo" ]
[ $(( split_hi + split_lo )) -eq 65 ]
[ "$split_lo" -lt "$split_hi" ]
grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE"
grep -q -- '--device "$DEVICES"' "$SERVE"
grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE"
@@ -144,9 +155,9 @@ print('draft-model wiring ok')
[ "$parallel" -ge 1 ]
[ "$batch" -ge "$ubatch" ]
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
# Per card, calibrated against rocm-smi at the deployed point (36:29, ctx
# 262144, batch 6144 x ubatch 1536, two slots, DFlash2), where the two cards
# measured 4.71 / 4.91 GiB free. The fit lands within ~0.05 GiB of that.
# Per card, calibrated against rocm-smi at the deployed point (35:30, ctx
# 262144, batch 6144 x ubatch 512, two slots, DFlash2), where the two cards
# measured 6.44 / 6.90 GiB free. The fit reproduces that to 0.01 GiB.
#
# card 0 = its share of the weights and of the KV (qwen35 spends KV on
# its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim
@@ -155,34 +166,44 @@ print('draft-model wiring ok')
# card 1 = the same for its own share, plus the WHOLE drafter: its weights
# (2.06e9 B), its KV -- BOUNDED BY ITS SLIDING WINDOW, so
# 20480 B/pos x 2048 positions rather than the context -- and
# ~2.7e6 B per ubatch unit of buffers on top of a ~4.8 GiB
# ~2.5e6 B per ubatch unit of buffers on top of a ~0.89 GiB
# constant. Its buffers are the heavier ones because it holds the
# output layers and the drafter's graphs.
#
# BOTH ubatch coefficients are measured at the deployed shape, from the
# 1536 -> 512 move: 1.30 GiB on card 0 and 2.41 GiB on card 1. That
# asymmetry is why shrinking the batch moved the balance toward card 1 and
# cost card 0 a layer -- the split and the batch cannot be chosen apart.
#
# The layer it charges for is the AVERAGE (0.70 GiB: 31.46e9 of weights plus
# 16 GiB of KV over 65 layers). This stack is a hybrid, so real steps vary
# around that -- 0.43 GiB for a DeltaNet layer, 1.43 GiB for a full-attention
# one -- which makes the drift check below CONSERVATIVE: it overstates the
# already trips it. That is the intent: moving the split means re-measuring,
# not re-arguing.
# one -- which makes the drift check below CONSERVATIVE: it overstates the gap
# a one-layer move creates, and so trips on moves a measurement might bless.
# That is the intent: moving the split means re-measuring, not re-arguing.
#
# The ubatch coefficients ARE fitted at 1536, so the floor check's "an
# 8192/2048 batch does not fit" is an EXTRAPOLATION, not a measurement --
# and it cannot see CONTIGUITY at all, which is the actual mechanism by which
# the DSpark-era 8192/2048 failed (a 2020 MiB pp buffer that found no room).
# The slack is one DeltaNet layer, rounded up. At this shape the balance
# cannot be tuned finer than one layer, so 35:30's 0.46 GiB gap IS the
# achievable optimum and a tighter slack would only reject it.
#
# What this model cannot see is CONTIGUITY -- the mechanism by which the
# DSpark-era 8192/2048 failed, a 2020 MiB pp buffer that found no room while
# every total looked fine. Total bytes are not the whole story for one big
# allocation.
local kv_pos=65536 draft_kv_pos=20480 floor slack
local card_hi card_lo diff
floor=$(( 1536 * 1024 * 1024 )) # 1.5 GiB per card
slack=$(( 512 * 1024 * 1024 )) # measurement tolerance
slack=$(( 512 * 1024 * 1024 )) # one DeltaNet layer, rounded up
card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \
+ kv_pos * ctx * split_hi / (split_hi + split_lo) \
+ 1442 * 1024 * ubatch ))
+ 381898377 \
+ 1363 * 1024 * ubatch ))
card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
+ DRAFT_BYTES \
+ kv_pos * ctx * split_lo / (split_hi + split_lo) \
+ draft_kv_pos * (ctx < 2048 ? ctx : 2048) \
+ 828697861 \
+ 2740 * 1024 * ubatch ))
+ 955046413 \
+ 2468 * 1024 * ubatch ))
[ "$card_hi" -le $(( CARD_BYTES - floor )) ]
[ "$card_lo" -le $(( CARD_BYTES - floor )) ]
diff=$(( card_hi - card_lo )); [ "$diff" -ge 0 ] || diff=$(( -diff ))