frank-medium: pay for DSpark's draft KV with a 38:27 split
DSpark declares no sliding window, so its draft context -- built with n_ctx = llama_n_ctx(ctx_tgt) -- caches the whole target window: 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB/position, i.e. 5.0 GiB at 262144, where DFlash2's sliding_window 2048 cost ~40 MiB. At a 1:1 split that put ROCm1 at 33.9 GB of 34.2 GB and the first draft graph capture died inside common_speculative_impl_draft_dflash::process with "ROCm error: out of memory" -- after the model had loaded and /health had answered, which is why the deploy looked like a crash-loop rather than a sizing failure. Handing three target layers to ROCm0 restores headroom on both cards at the full window. Measured at 262144: 28.52 GB / 30.71 GB, i.e. 5.30 GiB and 3.26 GiB free (was 2.7 GB free on the draft-carrying card with 1:1 + DFlash2). The unit seam now models the busier card as a skewed share plus the drafter's full-context KV, calibrated to the rocm-smi figure. AGENTS.md gains the durable lesson: read sliding_window out of the drafter's GGUF header before sizing a spec-decode deployment.
This commit is contained in:
@@ -41,7 +41,9 @@
|
||||
#
|
||||
# Drafter: magnitudedev/Qwen3.8-27B-DSpark-GGUF @ Q8_0, 1.27 GiB: 5 layers,
|
||||
# hidden 5120, block-diffusion proposal with the Markov head (dspark).
|
||||
# Decoding stays lossless -- the target verifies every drafted token.
|
||||
# Decoding stays lossless -- the target verifies every drafted token. Its KV
|
||||
# spans the whole context (see SIZING): unlike DFlash2 it declares no sliding
|
||||
# window, and that is what the tensor split below pays for.
|
||||
#
|
||||
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
|
||||
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
|
||||
@@ -51,12 +53,13 @@
|
||||
# state at any depth.
|
||||
#
|
||||
# SIZING, measured (rocm-smi, 2026-09-16). Weights split across the two cards;
|
||||
# the drafter rides the second. The loader's compute buffers are part of what
|
||||
# is seen below -- they are allocated at load, not per request:
|
||||
# the drafter AND its KV ride the second. The loader's compute buffers are part
|
||||
# of what is seen below -- they are allocated at load, not per request:
|
||||
#
|
||||
# CTX_SIZE GPU 0 GPU 1 headroom on the busier card
|
||||
# 131072 19.89 GB 24.30 GB 9.9 GB / 31 %
|
||||
# 262144 24.72 GB 29.14 GB 5.07 GB / 15 % <- deployed
|
||||
# CTX_SIZE split GPU 0 GPU 1 headroom on the busier card
|
||||
# 131072 1,1 19.89 GB 24.30 GB 9.9 GB / 31 % (DFlash2)
|
||||
# 262144 1,1 24.72 GB 29.14 GB 5.07 GB / 15 % (DFlash2) <- before
|
||||
# 262144 38,27 28.52 GB 30.71 GB 3.50 GB / 10 % (DSpark) <- deployed
|
||||
#
|
||||
# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
|
||||
# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
|
||||
@@ -64,6 +67,16 @@
|
||||
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
|
||||
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
|
||||
#
|
||||
# WHY THE SPLIT IS 38:27. The draft context is built with `n_ctx =
|
||||
# llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so the drafter's KV scales
|
||||
# with the target window, and DSpark -- unlike DFlash2, which declared
|
||||
# sliding_window 2048 and cost ~40 MiB -- caches all of it: 5 layers x 2 x 8 kv
|
||||
# heads x 128 dim x 2 B = 20 KiB per position, i.e. 5.0 GiB at 262144. At a
|
||||
# 1:1 split that put ROCm1 at 33.9 GB of 34.2 GB and the draft's first graph
|
||||
# capture died with `ROCm error: out of memory` (in
|
||||
# common_speculative_impl_draft_dflash::process). Handing three target layers
|
||||
# to ROCm0 buys back ~3 GiB of headroom on both cards at the full window.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
||||
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
||||
# as flags only.
|
||||
@@ -88,8 +101,11 @@ LISTEN_PORT=7770
|
||||
HIP_VISIBLE_DEVICES=0,1
|
||||
DEVICES=ROCm0,ROCm1
|
||||
SPLIT_MODE=layer
|
||||
# Empty = distribute layers by free VRAM (equal here, so 1,1 by hand).
|
||||
TENSOR_SPLIT=1,1
|
||||
|
||||
# Skewed 38:27 rather than 1:1: ROCm1 also carries the drafter and the whole of
|
||||
# its KV, so it must hold fewer target layers for the cards to end level. See
|
||||
# SIZING above -- this is measured, not computed.
|
||||
TENSOR_SPLIT=38,27
|
||||
|
||||
NGL=999
|
||||
# Auto-fit has no business resizing a context we set explicitly.
|
||||
|
||||
Reference in New Issue
Block a user