frank-medium: shift the layer split to 35:30, leaning work onto the x16 card

The two R9700s sit on unequal PCIe paths: ROCm0 (03:00.0) is CPU-root-port
Gen4 x16 (~31.5 GB/s); ROCm1 (0b:00.0) rides the B650 chipset through
bridges negotiating Gen4 x1 (~1.97 GB/s).  With the MTP drafter, output
layers and heavier compute buffers pinned to the last-layers card, live
rocm-smi read 98% busy on the x1 card vs 40% on the x16 card.  Moving one
DeltaNet layer (34) to ROCm0 shifts ~0.43 GiB off the bottlenecked card;
the seam fit lands 30.59/30.01 with gap 0.58 GiB, inside floor and slack.
36:29 fails both gates, so this lever is spent.
This commit is contained in:
Frank
2026-09-17 18:58:06 -07:00
parent d6bc245934
commit 6569c63ec4
4 changed files with 83 additions and 25 deletions
+24 -6
View File
@@ -141,12 +141,30 @@ HIP_VISIBLE_DEVICES=0,1
DEVICES=ROCm0,ROCm1
SPLIT_MODE=layer
# 34:31 -- the balance point at the 393216 pool (fit gap 1.44 GiB; MTP's head
# and its KV ride ROCm1 with the last layers, so the split sends MORE layers
# to ROCm0 than the 36:29 of the DFlash2/327680 configuration). One -ts step
# is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so do not
# move it without re-measuring.
TENSOR_SPLIT=34,31
# 35:30 -- operator call 2026-09-17: lean the work onto the x16 card. The
# two R9700s do NOT have symmetric PCIe paths. ROCm0 (03:00.0) hangs off
# CPU root port 00:01.1 at Gen4 x16 end to end (~31.5 GB/s); ROCm1
# (0b:00.0) rides the B650 chipset through bridges that each negotiate
# Gen4 x1 (~1.97 GB/s). Live rocm-smi showed the x1 card at 98% busy
# against 40% on the x16 card: the MTP drafter, the output layers and the
# heavier compute buffers are structurally pinned to the card holding the
# LAST layers (the draft context shares the target's weights there), so
# the bottlenecked card was doing most of the work. Moving one layer to
# ROCm0 shifts ~0.43 GiB of DeltaNet weights (layer 34 is a DeltaNet --
# the 4k..4k+2 position of block 8) off the x1 card and gives the healthy
# card the bigger share. The layer-split boundary itself is not the cost
# (~5.2 MB of activation per 512-token ubatch rides the wire), but the
# card behind the x1 link should not be the saturated one while its
# x16-sibling idles.
#
# WHY NOT FURTHER. One -ts step is one LAYER: ~0.43 GiB (DeltaNet) or
# ~1.43 GiB (attention). The next step, 36:29, would cross the
# full-attention layer at 35: the seam fit puts card 0 at 31.41 GiB
# (free 0.45 GiB, under the 1.0 GiB floor) with a 2.22 GiB gap (over the
# 1.483 GiB slack) -- both gates fail. 35:30 (fit: 30.59 / 30.01, gap
# 0.58) is the farthest safe shift toward the healthy card at this pool.
# Do not move it without re-measuring.
TENSOR_SPLIT=35,30
NGL=999
# Auto-fit has no business resizing a context we set explicitly.