frank-medium: 3x batch over 2 slots, and rebalance the split to pay for it
The operator's revised spec: batch x3 (6144/1536), parallel 2, a 1.5 GiB per-card floor, and the context maximised inside it. Measured at 262144 with the old 38:27 split, 3x batch was brutally asymmetric: ROCm1 (output layers + the drafter's graphs) paid ~3.3 GiB where ROCm0 paid ~1.7, leaving the draft-carrying card at 0.08 GiB free against ROCm0's 3.78. One -ts step is worth ~1.5 GiB at this batch, so the balance is a measurement: 41:24 overshot (1.47 / 2.38), 40:25 landed at 1.92 / 1.96 GiB free. That balance is also what maximises the context -- an unbalanced split would have to buy its headroom out of CTX_SIZE, and CTX_SIZE is already at the model's trained window (262144), now split over two slots of 131072. Verified under two concurrent 1832-token prompts: 200/200, 1.87 / 1.86 GiB free, n_slots = 2, n_ctx_slot = 131072, tool calls intact, and DSpark drafting on BOTH slots (acceptance 0.38/0.22 and 0.52/0.43). 8192/2048 does not fit at this context at all: its 2020 MiB pp buffer fails to allocate on ROCm1, and 4096/1024 fits but leaves 1.11 GiB -- both recorded in the env's sizing table. The unit seam now models both cards (share + draft + ubatch-scaled buffers, calibrated to the rocm-smi figures to within 0.002%), asserts the 1.5 GiB floor on each, and asserts the two cards stay within 0.5 GiB of each other. Both assertions were confirmed to FAIL on a deliberately bad config (ubatch 8192, and a 1:1 split) rather than pass by construction.
This commit is contained in:
Frank committed
1 parent
88f0d79021
commit
b312042706
3 files changed
+104
-60
No files matched your search
@@ -47,7 +47,7 @@ DRAFT_BYTES=1359284737 # DSpark Q8_0 drafter
|
||||
@test "both R9700s serve ONE model, layer-split inside one process" {
|
||||
grep -q '^SPLIT_MODE=layer$' "$ENV_FILE"
|
||||
grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE"
|
||||
grep -q '^TENSOR_SPLIT=38,27$' "$ENV_FILE"
|
||||
grep -q '^TENSOR_SPLIT=40,25$' "$ENV_FILE"
|
||||
grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE"
|
||||
grep -q -- '--device "$DEVICES"' "$SERVE"
|
||||
grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE"
|
||||
@@ -129,37 +129,57 @@ print('draft-model wiring ok')
|
||||
grep -q 'verifying sizes' "$FETCH"
|
||||
}
|
||||
|
||||
@test "one slot's KV still fits a card beside the weights that card holds" {
|
||||
local ctx parallel cache_k cache_v split_hi split_lo
|
||||
@test "both cards stay balanced and keep the operator's 1.5 GiB floor" {
|
||||
local ctx parallel cache_k cache_v split_hi split_lo batch ubatch
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")"
|
||||
batch="$(sed -nE 's/^BATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
ubatch="$(sed -nE 's/^UBATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ]
|
||||
[ -n "$split_hi" ] && [ -n "$split_lo" ]
|
||||
[ -n "$split_hi" ] && [ -n "$split_lo" ] && [ -n "$batch" ] && [ -n "$ubatch" ]
|
||||
[ "$parallel" -ge 1 ]
|
||||
[ "$batch" -ge "$ubatch" ]
|
||||
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
|
||||
# qwen35 spends KV on its 16 full-attention layers only: 2 (K,V) x 4 heads
|
||||
# x 256 dim x 2 B = 64 KiB per token at f16, carried by whichever card a
|
||||
# layer split puts those layers on. The busier card is the SECOND one: its
|
||||
# share of the weights and KV, plus the WHOLE drafter -- whose KV spans the
|
||||
# full context, because DSpark declares no sliding window (5 x 2 x 8 x 128
|
||||
# x 2 B = 20480 B per position) -- plus the loader's compute buffers.
|
||||
# Per card, calibrated against rocm-smi at the deployed point (40:25, ctx
|
||||
# 262144, batch 6144 x ubatch 1536, two slots, DSpark) where the two cards
|
||||
# measured 32.199e9 / 32.212e9 B used:
|
||||
#
|
||||
# This model is deliberately calibrated against rocm-smi: at 262144 / 38:27
|
||||
# it predicts 30.69e9 B against the 30.71e9 B measured on the busier card
|
||||
# (2026-09-16). Keep it that way -- a model that is pessimistic beyond a few
|
||||
# GiB would reject a context that fits.
|
||||
local per_card bufs
|
||||
bufs=$(( 3584 * 1024 * 1024 ))
|
||||
per_card=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
|
||||
+ DRAFT_BYTES \
|
||||
+ (ctx * 65536) * split_lo / (split_hi + split_lo) \
|
||||
+ ctx * 20480 \
|
||||
+ bufs ))
|
||||
[ "$per_card" -le "$CARD_BYTES" ]
|
||||
# card 0 = its share of the weights and of the KV (qwen35 spends KV on
|
||||
# its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim
|
||||
# x 2 B = 64 KiB per token at f16) plus ~1.4e6 B per ubatch unit
|
||||
# of compute buffers
|
||||
# card 1 = the same for its own share, plus the WHOLE drafter -- its
|
||||
# weights AND its full-context KV, because DSpark declares no
|
||||
# sliding window (5 x 2 x 8 x 128 x 2 B = 20480 B per position;
|
||||
# the draft context is sized from the target's:
|
||||
# `n_ctx = llama_n_ctx(ctx_tgt)`) -- and ~2.7e6 B per ubatch unit
|
||||
# on top of a ~2.3 GiB constant. Its buffers are the heavier ones
|
||||
# because it holds the output layers and the drafter's graphs.
|
||||
#
|
||||
# Both cards must clear the operator's floor, and a change to ctx, ubatch or
|
||||
# the split has to keep them from drifting apart -- an unbalanced split is
|
||||
# what cost ROCm1 3.7 GiB of headroom when the batch tripled (2026-09-16).
|
||||
local kv_pos=65536 draft_kv_pos=20480 floor slack
|
||||
local card_hi card_lo diff
|
||||
floor=$(( 1536 * 1024 * 1024 )) # 1.5 GiB per card
|
||||
slack=$(( 512 * 1024 * 1024 )) # measurement tolerance
|
||||
card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \
|
||||
+ kv_pos * ctx * split_hi / (split_hi + split_lo) \
|
||||
+ 1442 * 1024 * ubatch ))
|
||||
card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
|
||||
+ DRAFT_BYTES \
|
||||
+ kv_pos * ctx * split_lo / (split_hi + split_lo) \
|
||||
+ draft_kv_pos * ctx \
|
||||
+ 2353 * 1024 * 1024 \
|
||||
+ 2740 * 1024 * ubatch ))
|
||||
[ "$card_hi" -le $(( CARD_BYTES - floor )) ]
|
||||
[ "$card_lo" -le $(( CARD_BYTES - floor )) ]
|
||||
diff=$(( card_hi - card_lo )); [ "$diff" -ge 0 ] || diff=$(( -diff ))
|
||||
[ "$diff" -le "$slack" ]
|
||||
# ...and the window may not exceed what the checkpoint was trained for.
|
||||
[ "$ctx" -le 262144 ]
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user