frank-medium: step 3 -- the skew is the balance, and equality is off-limits
The 38:27 skew folded into the DSpark swap already does what step 3 asks: the draft-carrying card sits 2.08 GiB above the other (5.49 / 3.41 GiB free) instead of the ~9 GiB a 1:1 split would leave. Chasing an exact balance is a mistake worth recording: 39:26 measured tighter (3.86 / 4.69 GiB free) but that server died on its eighth request with SDMA0 UTCL2 page faults in the amdgpu driver -- "ROCm error: an illegal memory access was encountered" -> ggml_abort, in the cross-card boundary copy that is the layer split's hot path. 38:27 has since served the whole e2e gate and 12 varied requests with no fault, so the residual 2.08 GiB gap is the safe side of this trade. Comment-only: no restart needed.
This commit is contained in:
1 parent
169eba0734
commit
ad18d2ddd8
1 file changed
+14
-2
@@ -77,6 +77,16 @@
|
|||||||
# common_speculative_impl_draft_dflash::process). Handing three target layers
|
# common_speculative_impl_draft_dflash::process). Handing three target layers
|
||||||
# to ROCm0 buys back ~3 GiB of headroom on both cards at the full window.
|
# to ROCm0 buys back ~3 GiB of headroom on both cards at the full window.
|
||||||
#
|
#
|
||||||
|
# EQUALITY WAS TRIED AND REJECTED. 39:26 levels the cards more closely (3.86 /
|
||||||
|
# 4.69 GiB free -- an 0.83 GiB gap, against 2.08 GiB here), but that server
|
||||||
|
# died on its eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults
|
||||||
|
# and the engine surfaced `ROCm error: an illegal memory access was
|
||||||
|
# encountered` -> ggml_abort, i.e. a fault in the cross-card boundary copy,
|
||||||
|
# which is this layer split's hot path. 38:27 has since served the whole e2e
|
||||||
|
# gate plus a burst of varied requests with none. Do not chase an exact
|
||||||
|
# balance: take the skew that fits and leave the drafter's card on the safe
|
||||||
|
# side of the gap.
|
||||||
|
#
|
||||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
||||||
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
||||||
# as flags only.
|
# as flags only.
|
||||||
@@ -103,8 +113,10 @@ DEVICES=ROCm0,ROCm1
|
|||||||
SPLIT_MODE=layer
|
SPLIT_MODE=layer
|
||||||
|
|
||||||
# Skewed 38:27 rather than 1:1: ROCm1 also carries the drafter and the whole of
|
# Skewed 38:27 rather than 1:1: ROCm1 also carries the drafter and the whole of
|
||||||
# its KV, so it must hold fewer target layers for the cards to end level. See
|
# its KV, so it must hold fewer target layers for the cards to end level -- at
|
||||||
# SIZING above -- this is measured, not computed.
|
# 1:1 the drafter's card would sit ~9 GiB above the other. This is the measured
|
||||||
|
# point that keeps BOTH cards above a 3 GiB floor at the full window; a tighter
|
||||||
|
# balance was tried and rejected (see SIZING above).
|
||||||
TENSOR_SPLIT=38,27
|
TENSOR_SPLIT=38,27
|
||||||
|
|
||||||
NGL=999
|
NGL=999
|
||||||
|
|||||||
Reference in new issue
Block a user