frank-medium: step 3 -- the skew is the balance, and equality is off-limits

The 38:27 skew folded into the DSpark swap already does what step 3 asks: the
draft-carrying card sits 2.08 GiB above the other (5.49 / 3.41 GiB free) instead
of the ~9 GiB a 1:1 split would leave.

Chasing an exact balance is a mistake worth recording: 39:26 measured tighter
(3.86 / 4.69 GiB free) but that server died on its eighth request with SDMA0
UTCL2 page faults in the amdgpu driver -- "ROCm error: an illegal memory access
was encountered" -> ggml_abort, in the cross-card boundary copy that is the
layer split's hot path. 38:27 has since served the whole e2e gate and 12 varied
requests with no fault, so the residual 2.08 GiB gap is the safe side of this
trade. Comment-only: no restart needed.
This commit is contained in:
Frank
2026-09-16 20:00:17 -07:00
parent 169eba0734
commit ad18d2ddd8
+14 -2
View File
@@ -77,6 +77,16 @@
# common_speculative_impl_draft_dflash::process). Handing three target layers
# to ROCm0 buys back ~3 GiB of headroom on both cards at the full window.
#
# EQUALITY WAS TRIED AND REJECTED. 39:26 levels the cards more closely (3.86 /
# 4.69 GiB free -- an 0.83 GiB gap, against 2.08 GiB here), but that server
# died on its eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults
# and the engine surfaced `ROCm error: an illegal memory access was
# encountered` -> ggml_abort, i.e. a fault in the cross-card boundary copy,
# which is this layer split's hot path. 38:27 has since served the whole e2e
# gate plus a burst of varied requests with none. Do not chase an exact
# balance: take the skew that fits and leave the drafter's card on the safe
# side of the gap.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
# as flags only.
@@ -103,8 +113,10 @@ DEVICES=ROCm0,ROCm1
SPLIT_MODE=layer
# Skewed 38:27 rather than 1:1: ROCm1 also carries the drafter and the whole of
# its KV, so it must hold fewer target layers for the cards to end level. See
# SIZING above -- this is measured, not computed.
# its KV, so it must hold fewer target layers for the cards to end level -- at
# 1:1 the drafter's card would sit ~9 GiB above the other. This is the measured
# point that keeps BOTH cards above a 3 GiB floor at the full window; a tighter
# balance was tried and rejected (see SIZING above).
TENSOR_SPLIT=38,27
NGL=999