frank-medium: record 2026-09-18 rebalance measurement, keep 35:30

Isolated-port (:7790) 2k-token decode with AVG rocm-smi util over the
request (peak util lies: a layer split touches both cards every token):

  35:30 draft ROCm1  31.52 t/s  util0 41%  util1 98%  <- deployed
  36:29 draft ROCm1  31.49 t/s  util0 43%  util1 98%
  35:30 draft ROCm0  31.52 t/s  util0 42%  util1 98%
  36:29 draft ROCm0  31.48 t/s  util0 43%  util1 98%

The x1 card (ROCm1) stays the 98% bottleneck in every configuration.
Moving the MTP drafter to the x16 card does not flip the balance: the
per-step cost is the verify phase (both cards run the full target) plus
the logits, which are pinned to the card holding the last layers. A
target-layer move (36:29) buys 2 points on the x16 card and erodes its
VRAM margin (fit free 0.45 GiB) for no throughput change. The x1 card is
compute-bound, not PCIe-bound (~5.2 MB activation per ubatch on the link),
so saturating it costs no tokens. 35:30 is the balance; no split change.
This commit is contained in:
Frank
2026-09-18 00:28:29 -07:00
parent 9764e86986
commit 6c6464a31a
+20
View File
@@ -164,6 +164,26 @@ SPLIT_MODE=layer
# 1.483 GiB slack) -- both gates fail. 35:30 (fit: 30.59 / 30.01, gap
# 0.58) is the farthest safe shift toward the healthy card at this pool.
# Do not move it without re-measuring.
# MEASURED 2026-09-18 (isolated :7790, 2k-token decode, AVG rocm-smi util over
# the request -- peak util lies, a layer split touches both cards every token):
#
# split draft_dev decode AVG util0(x16) AVG util1(x1)
# 35,30 ROCm1 31.52 41 % 98 % <- deployed
# 36,29 ROCm1 31.49 43 % 98 %
# 35,30 ROCm0 31.52 42 % 98 %
# 36,29 ROCm0 31.48 43 % 98 %
#
# The x1 card stays the 98 % bottleneck in EVERY configuration. The lever that
# was tried is moving the MTP drafter to the x16 card (SPEC_DRAFT_DEVICE=ROCm0):
# it does NOT flip the balance, because the drafter is a small nextn head and the
# per-step cost is dominated by the VERIFY phase -- both cards run the full target
# -- plus the LOGITS, which are pinned to the card holding the last layers. A
# target-layer move (36:29) buys 2 points on the x16 card and erodes its VRAM
# margin (fit: free 0.45 GiB) for nothing. The x1 card is COMPUTE-bound, not
# PCIe-bound (~5.2 MB of activation per 512-token ubatch crosses the link);
# saturating it costs no tokens beyond what any R9700 would. 35:30 is the
# balance; do not chase the x16 card past it.
TENSOR_SPLIT=35,30
NGL=999