frank-medium: ubatch 512, and rebalance the split to 35:30
The batch/ubatch sweep (scripts/bench-batch-ubatch.sh, 23 cells of a 28-cell grid, 3 reps each) says the ubatch knob has an OPTIMUM, not a plateau: ubatch 256 384 512 768 1024 1280 1536 1792 2048 tok/s 1046 1068 1193 1135 1152 1151 1165 1148 1078 The curve is remarkably reproducible -- 512 came out 1192.88 / 1193.23 / 1193.28 / 1193.88 across batch 2048, 4096, 6144 and 8192, and 2048 came out 1077.38 / 1077.54 / 1077.59 -- so this is a deterministic property of the shape, not noise. The logical batch is throughput-neutral (2048..8192 all within 2 % at a fixed ubatch) and VRAM-neutral (the compute buffers are sized by ubatch alone). Decode is 15.88 tok/s at depth 65536 on every single cell. So the deployed 6144x1536 was both 2.4 % slower than the optimum and 3.71 GiB more expensive. Measured at the deployed shape (262144, 2 slots, DFlash2): 36,29 ubatch 1536 -> 4.71 / 4.91 GiB free (what was deployed) 36,29 ubatch 512 -> 6.01 / 7.32 GiB free (ubatch alone) 35,30 ubatch 512 -> 6.44 / 6.90 GiB free <- deployed The ubatch move freed 1.30 GiB on ROCm0 but 2.41 GiB on ROCm1, because ROCm1 holds the output layers and the drafter's graphs -- so it shifted the balance toward ROCm1 by 1.31 GiB and the split had to follow it one layer. 0.46 GiB is one DeltaNet layer's worth of asymmetry, i.e. the granularity floor: no split can be tuned finer than a layer, which is why 34,31 is not better. Re-fitted the seam with both ubatch coefficients measured at this shape and re-anchored it on the new deployed point (it reproduces 6.44 / 6.90 to 0.01 GiB). Every neighbouring split still fails it: 36,29 -> 1.85, 34,31 -> 0.94, 37,28 -> 3.24, 1,1 -> 3.03, 39,26 -> 6.03 GiB. Also replaced the unit test's literal `TENSOR_SPLIT=36,29` assertion with the shape it actually cares about -- two shares that add up to the model's 65 blocks, with ROCm1 taking the fewer -- so re-measuring the balance no longer breaks a test about parallelism. The numeric contract lives in the seam test.
This commit is contained in:
Frank
@@ -47,7 +47,18 @@ DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter
|
||||
@test "both R9700s serve ONE model, layer-split inside one process" {
|
||||
grep -q '^SPLIT_MODE=layer$' "$ENV_FILE"
|
||||
grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE"
|
||||
grep -q '^TENSOR_SPLIT=36,29$' "$ENV_FILE"
|
||||
# The split is a layer PAIR: the shares add up to the model's 65 blocks (one
|
||||
# -ts unit per layer) and ROCm1 -- the card that also carries the drafter and
|
||||
# the output layers -- takes the FEWER of the two. WHICH pair balances the
|
||||
# cards is the seam test's job, which recomputes it from this file; pinning a
|
||||
# literal here would only mean that re-measuring the balance breaks a test
|
||||
# about parallelism.
|
||||
local split_hi split_lo
|
||||
split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")"
|
||||
[ -n "$split_hi" ] && [ -n "$split_lo" ]
|
||||
[ $(( split_hi + split_lo )) -eq 65 ]
|
||||
[ "$split_lo" -lt "$split_hi" ]
|
||||
grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE"
|
||||
grep -q -- '--device "$DEVICES"' "$SERVE"
|
||||
grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE"
|
||||
@@ -144,9 +155,9 @@ print('draft-model wiring ok')
|
||||
[ "$parallel" -ge 1 ]
|
||||
[ "$batch" -ge "$ubatch" ]
|
||||
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
|
||||
# Per card, calibrated against rocm-smi at the deployed point (36:29, ctx
|
||||
# 262144, batch 6144 x ubatch 1536, two slots, DFlash2), where the two cards
|
||||
# measured 4.71 / 4.91 GiB free. The fit lands within ~0.05 GiB of that.
|
||||
# Per card, calibrated against rocm-smi at the deployed point (35:30, ctx
|
||||
# 262144, batch 6144 x ubatch 512, two slots, DFlash2), where the two cards
|
||||
# measured 6.44 / 6.90 GiB free. The fit reproduces that to 0.01 GiB.
|
||||
#
|
||||
# card 0 = its share of the weights and of the KV (qwen35 spends KV on
|
||||
# its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim
|
||||
@@ -155,34 +166,44 @@ print('draft-model wiring ok')
|
||||
# card 1 = the same for its own share, plus the WHOLE drafter: its weights
|
||||
# (2.06e9 B), its KV -- BOUNDED BY ITS SLIDING WINDOW, so
|
||||
# 20480 B/pos x 2048 positions rather than the context -- and
|
||||
# ~2.7e6 B per ubatch unit of buffers on top of a ~4.8 GiB
|
||||
# ~2.5e6 B per ubatch unit of buffers on top of a ~0.89 GiB
|
||||
# constant. Its buffers are the heavier ones because it holds the
|
||||
# output layers and the drafter's graphs.
|
||||
#
|
||||
# BOTH ubatch coefficients are measured at the deployed shape, from the
|
||||
# 1536 -> 512 move: 1.30 GiB on card 0 and 2.41 GiB on card 1. That
|
||||
# asymmetry is why shrinking the batch moved the balance toward card 1 and
|
||||
# cost card 0 a layer -- the split and the batch cannot be chosen apart.
|
||||
#
|
||||
# The layer it charges for is the AVERAGE (0.70 GiB: 31.46e9 of weights plus
|
||||
# 16 GiB of KV over 65 layers). This stack is a hybrid, so real steps vary
|
||||
# around that -- 0.43 GiB for a DeltaNet layer, 1.43 GiB for a full-attention
|
||||
# one -- which makes the drift check below CONSERVATIVE: it overstates the
|
||||
# already trips it. That is the intent: moving the split means re-measuring,
|
||||
# not re-arguing.
|
||||
# one -- which makes the drift check below CONSERVATIVE: it overstates the gap
|
||||
# a one-layer move creates, and so trips on moves a measurement might bless.
|
||||
# That is the intent: moving the split means re-measuring, not re-arguing.
|
||||
#
|
||||
# The ubatch coefficients ARE fitted at 1536, so the floor check's "an
|
||||
# 8192/2048 batch does not fit" is an EXTRAPOLATION, not a measurement --
|
||||
# and it cannot see CONTIGUITY at all, which is the actual mechanism by which
|
||||
# the DSpark-era 8192/2048 failed (a 2020 MiB pp buffer that found no room).
|
||||
# The slack is one DeltaNet layer, rounded up. At this shape the balance
|
||||
# cannot be tuned finer than one layer, so 35:30's 0.46 GiB gap IS the
|
||||
# achievable optimum and a tighter slack would only reject it.
|
||||
#
|
||||
# What this model cannot see is CONTIGUITY -- the mechanism by which the
|
||||
# DSpark-era 8192/2048 failed, a 2020 MiB pp buffer that found no room while
|
||||
# every total looked fine. Total bytes are not the whole story for one big
|
||||
# allocation.
|
||||
local kv_pos=65536 draft_kv_pos=20480 floor slack
|
||||
local card_hi card_lo diff
|
||||
floor=$(( 1536 * 1024 * 1024 )) # 1.5 GiB per card
|
||||
slack=$(( 512 * 1024 * 1024 )) # measurement tolerance
|
||||
slack=$(( 512 * 1024 * 1024 )) # one DeltaNet layer, rounded up
|
||||
card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \
|
||||
+ kv_pos * ctx * split_hi / (split_hi + split_lo) \
|
||||
+ 1442 * 1024 * ubatch ))
|
||||
+ 381898377 \
|
||||
+ 1363 * 1024 * ubatch ))
|
||||
card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
|
||||
+ DRAFT_BYTES \
|
||||
+ kv_pos * ctx * split_lo / (split_hi + split_lo) \
|
||||
+ draft_kv_pos * (ctx < 2048 ? ctx : 2048) \
|
||||
+ 828697861 \
|
||||
+ 2740 * 1024 * ubatch ))
|
||||
+ 955046413 \
|
||||
+ 2468 * 1024 * ubatch ))
|
||||
[ "$card_hi" -le $(( CARD_BYTES - floor )) ]
|
||||
[ "$card_lo" -le $(( CARD_BYTES - floor )) ]
|
||||
diff=$(( card_hi - card_lo )); [ "$diff" -ge 0 ] || diff=$(( -diff ))
|
||||
|
||||
Reference in New Issue
Block a user