frank-medium: record 2026-09-18 rebalance measurement, keep 35:30
Isolated-port (:7790) 2k-token decode with AVG rocm-smi util over the request (peak util lies: a layer split touches both cards every token): 35:30 draft ROCm1 31.52 t/s util0 41% util1 98% <- deployed 36:29 draft ROCm1 31.49 t/s util0 43% util1 98% 35:30 draft ROCm0 31.52 t/s util0 42% util1 98% 36:29 draft ROCm0 31.48 t/s util0 43% util1 98% The x1 card (ROCm1) stays the 98% bottleneck in every configuration. Moving the MTP drafter to the x16 card does not flip the balance: the per-step cost is the verify phase (both cards run the full target) plus the logits, which are pinned to the card holding the last layers. A target-layer move (36:29) buys 2 points on the x16 card and erodes its VRAM margin (fit free 0.45 GiB) for no throughput change. The x1 card is compute-bound, not PCIe-bound (~5.2 MB activation per ubatch on the link), so saturating it costs no tokens. 35:30 is the balance; no split change.
This commit is contained in:
@@ -164,6 +164,26 @@ SPLIT_MODE=layer
|
||||
# 1.483 GiB slack) -- both gates fail. 35:30 (fit: 30.59 / 30.01, gap
|
||||
# 0.58) is the farthest safe shift toward the healthy card at this pool.
|
||||
# Do not move it without re-measuring.
|
||||
|
||||
# MEASURED 2026-09-18 (isolated :7790, 2k-token decode, AVG rocm-smi util over
|
||||
# the request -- peak util lies, a layer split touches both cards every token):
|
||||
#
|
||||
# split draft_dev decode AVG util0(x16) AVG util1(x1)
|
||||
# 35,30 ROCm1 31.52 41 % 98 % <- deployed
|
||||
# 36,29 ROCm1 31.49 43 % 98 %
|
||||
# 35,30 ROCm0 31.52 42 % 98 %
|
||||
# 36,29 ROCm0 31.48 43 % 98 %
|
||||
#
|
||||
# The x1 card stays the 98 % bottleneck in EVERY configuration. The lever that
|
||||
# was tried is moving the MTP drafter to the x16 card (SPEC_DRAFT_DEVICE=ROCm0):
|
||||
# it does NOT flip the balance, because the drafter is a small nextn head and the
|
||||
# per-step cost is dominated by the VERIFY phase -- both cards run the full target
|
||||
# -- plus the LOGITS, which are pinned to the card holding the last layers. A
|
||||
# target-layer move (36:29) buys 2 points on the x16 card and erodes its VRAM
|
||||
# margin (fit: free 0.45 GiB) for nothing. The x1 card is COMPUTE-bound, not
|
||||
# PCIe-bound (~5.2 MB of activation per 512-token ubatch crosses the link);
|
||||
# saturating it costs no tokens beyond what any R9700 would. 35:30 is the
|
||||
# balance; do not chase the x16 card past it.
|
||||
TENSOR_SPLIT=35,30
|
||||
|
||||
NGL=999
|
||||
|
||||
Reference in New Issue
Block a user