frank-medium: record 2026-09-18 rebalance measurement, keep 35:30
Isolated-port (:7790) 2k-token decode with AVG rocm-smi util over the request (peak util lies: a layer split touches both cards every token): 35:30 draft ROCm1 31.52 t/s util0 41% util1 98% <- deployed 36:29 draft ROCm1 31.49 t/s util0 43% util1 98% 35:30 draft ROCm0 31.52 t/s util0 42% util1 98% 36:29 draft ROCm0 31.48 t/s util0 43% util1 98% The x1 card (ROCm1) stays the 98% bottleneck in every configuration. Moving the MTP drafter to the x16 card does not flip the balance: the per-step cost is the verify phase (both cards run the full target) plus the logits, which are pinned to the card holding the last layers. A target-layer move (36:29) buys 2 points on the x16 card and erodes its VRAM margin (fit free 0.45 GiB) for no throughput change. The x1 card is compute-bound, not PCIe-bound (~5.2 MB activation per ubatch on the link), so saturating it costs no tokens. 35:30 is the balance; no split change.
This commit is contained in:
1 parent
9764e86986
commit
6c6464a31a
1 file changed
+20
@@ -164,6 +164,26 @@ SPLIT_MODE=layer
|
|||||||
# 1.483 GiB slack) -- both gates fail. 35:30 (fit: 30.59 / 30.01, gap
|
# 1.483 GiB slack) -- both gates fail. 35:30 (fit: 30.59 / 30.01, gap
|
||||||
# 0.58) is the farthest safe shift toward the healthy card at this pool.
|
# 0.58) is the farthest safe shift toward the healthy card at this pool.
|
||||||
# Do not move it without re-measuring.
|
# Do not move it without re-measuring.
|
||||||
|
|
||||||
|
# MEASURED 2026-09-18 (isolated :7790, 2k-token decode, AVG rocm-smi util over
|
||||||
|
# the request -- peak util lies, a layer split touches both cards every token):
|
||||||
|
#
|
||||||
|
# split draft_dev decode AVG util0(x16) AVG util1(x1)
|
||||||
|
# 35,30 ROCm1 31.52 41 % 98 % <- deployed
|
||||||
|
# 36,29 ROCm1 31.49 43 % 98 %
|
||||||
|
# 35,30 ROCm0 31.52 42 % 98 %
|
||||||
|
# 36,29 ROCm0 31.48 43 % 98 %
|
||||||
|
#
|
||||||
|
# The x1 card stays the 98 % bottleneck in EVERY configuration. The lever that
|
||||||
|
# was tried is moving the MTP drafter to the x16 card (SPEC_DRAFT_DEVICE=ROCm0):
|
||||||
|
# it does NOT flip the balance, because the drafter is a small nextn head and the
|
||||||
|
# per-step cost is dominated by the VERIFY phase -- both cards run the full target
|
||||||
|
# -- plus the LOGITS, which are pinned to the card holding the last layers. A
|
||||||
|
# target-layer move (36:29) buys 2 points on the x16 card and erodes its VRAM
|
||||||
|
# margin (fit: free 0.45 GiB) for nothing. The x1 card is COMPUTE-bound, not
|
||||||
|
# PCIe-bound (~5.2 MB of activation per 512-token ubatch crosses the link);
|
||||||
|
# saturating it costs no tokens beyond what any R9700 would. 35:30 is the
|
||||||
|
# balance; do not chase the x16 card past it.
|
||||||
TENSOR_SPLIT=35,30
|
TENSOR_SPLIT=35,30
|
||||||
|
|
||||||
NGL=999
|
NGL=999
|
||||||
|
|||||||
Reference in new issue
Block a user