frank-medium: shift the layer split to 35:30, leaning work onto the x16 card
The two R9700s sit on unequal PCIe paths: ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s); ROCm1 (0b:00.0) rides the B650 chipset through bridges negotiating Gen4 x1 (~1.97 GB/s). With the MTP drafter, output layers and heavier compute buffers pinned to the last-layers card, live rocm-smi read 98% busy on the x1 card vs 40% on the x16 card. Moving one DeltaNet layer (34) to ROCm0 shifts ~0.43 GiB off the bottlenecked card; the seam fit lands 30.59/30.01 with gap 0.58 GiB, inside floor and slack. 36:29 fails both gates, so this lever is spent.
This commit is contained in:
@@ -141,12 +141,30 @@ HIP_VISIBLE_DEVICES=0,1
|
||||
DEVICES=ROCm0,ROCm1
|
||||
SPLIT_MODE=layer
|
||||
|
||||
# 34:31 -- the balance point at the 393216 pool (fit gap 1.44 GiB; MTP's head
|
||||
# and its KV ride ROCm1 with the last layers, so the split sends MORE layers
|
||||
# to ROCm0 than the 36:29 of the DFlash2/327680 configuration). One -ts step
|
||||
# is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so do not
|
||||
# move it without re-measuring.
|
||||
TENSOR_SPLIT=34,31
|
||||
# 35:30 -- operator call 2026-09-17: lean the work onto the x16 card. The
|
||||
# two R9700s do NOT have symmetric PCIe paths. ROCm0 (03:00.0) hangs off
|
||||
# CPU root port 00:01.1 at Gen4 x16 end to end (~31.5 GB/s); ROCm1
|
||||
# (0b:00.0) rides the B650 chipset through bridges that each negotiate
|
||||
# Gen4 x1 (~1.97 GB/s). Live rocm-smi showed the x1 card at 98% busy
|
||||
# against 40% on the x16 card: the MTP drafter, the output layers and the
|
||||
# heavier compute buffers are structurally pinned to the card holding the
|
||||
# LAST layers (the draft context shares the target's weights there), so
|
||||
# the bottlenecked card was doing most of the work. Moving one layer to
|
||||
# ROCm0 shifts ~0.43 GiB of DeltaNet weights (layer 34 is a DeltaNet --
|
||||
# the 4k..4k+2 position of block 8) off the x1 card and gives the healthy
|
||||
# card the bigger share. The layer-split boundary itself is not the cost
|
||||
# (~5.2 MB of activation per 512-token ubatch rides the wire), but the
|
||||
# card behind the x1 link should not be the saturated one while its
|
||||
# x16-sibling idles.
|
||||
#
|
||||
# WHY NOT FURTHER. One -ts step is one LAYER: ~0.43 GiB (DeltaNet) or
|
||||
# ~1.43 GiB (attention). The next step, 36:29, would cross the
|
||||
# full-attention layer at 35: the seam fit puts card 0 at 31.41 GiB
|
||||
# (free 0.45 GiB, under the 1.0 GiB floor) with a 2.22 GiB gap (over the
|
||||
# 1.483 GiB slack) -- both gates fail. 35:30 (fit: 30.59 / 30.01, gap
|
||||
# 0.58) is the farthest safe shift toward the healthy card at this pool.
|
||||
# Do not move it without re-measuring.
|
||||
TENSOR_SPLIT=35,30
|
||||
|
||||
NGL=999
|
||||
# Auto-fit has no business resizing a context we set explicitly.
|
||||
|
||||
Reference in New Issue
Block a user