frank-medium: 3x batch over 2 slots, and rebalance the split to pay for it
The operator's revised spec: batch x3 (6144/1536), parallel 2, a 1.5 GiB per-card floor, and the context maximised inside it. Measured at 262144 with the old 38:27 split, 3x batch was brutally asymmetric: ROCm1 (output layers + the drafter's graphs) paid ~3.3 GiB where ROCm0 paid ~1.7, leaving the draft-carrying card at 0.08 GiB free against ROCm0's 3.78. One -ts step is worth ~1.5 GiB at this batch, so the balance is a measurement: 41:24 overshot (1.47 / 2.38), 40:25 landed at 1.92 / 1.96 GiB free. That balance is also what maximises the context -- an unbalanced split would have to buy its headroom out of CTX_SIZE, and CTX_SIZE is already at the model's trained window (262144), now split over two slots of 131072. Verified under two concurrent 1832-token prompts: 200/200, 1.87 / 1.86 GiB free, n_slots = 2, n_ctx_slot = 131072, tool calls intact, and DSpark drafting on BOTH slots (acceptance 0.38/0.22 and 0.52/0.43). 8192/2048 does not fit at this context at all: its 2020 MiB pp buffer fails to allocate on ROCm1, and 4096/1024 fits but leaves 1.11 GiB -- both recorded in the env's sizing table. The unit seam now models both cards (share + draft + ubatch-scaled buffers, calibrated to the rocm-smi figures to within 0.002%), asserts the 1.5 GiB floor on each, and asserts the two cards stay within 0.5 GiB of each other. Both assertions were confirmed to FAIL on a deliberately bad config (ubatch 8192, and a 1:1 split) rather than pass by construction.
This commit is contained in:
Frank
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (38:27), 262144 ctx / 1 slot, text only | 7770 |
|
||||
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (40:25), 262144 ctx over 2 slots, batch 6144x1536, text only | 7770 |
|
||||
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
|
||||
|
||||
@@ -305,7 +305,9 @@ systemd unit `scripts/frank-medium.service`) serving
|
||||
`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
|
||||
(29.29 GiB) with the **magnitudedev DSpark** `Q8_0` drafter chained to the
|
||||
draftless **ngram-mod** matcher, in **one process across both R9700s**,
|
||||
`--split-mode layer` at a **38:27 tensor split**, 262144 ctx / 1 slot, text only.
|
||||
`--split-mode layer` at a **40:25 tensor split** (the balance point for the
|
||||
deployed batch: the drafter rides ROCm1, so it takes fewer target layers),
|
||||
262144 ctx over **2 slots** of 131072, batch 6144x1536, text only.
|
||||
|
||||
**Layer split is why both cards now serve ONE model.** Tensor parallelism is
|
||||
impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism —
|
||||
@@ -509,6 +511,15 @@ linked runbooks.
|
||||
drafter's GGUF header before sizing any spec-decode deployment, and prefer a
|
||||
skewed `-ts` over a context cut when the full window matters. See
|
||||
[[frank-medium-llamacpp-dflash2]].
|
||||
- **The compute buffers of a bigger `--ubatch-size` land ~4x heavier on the card
|
||||
that holds the output layers and the drafter.** On frank-medium (2026-09-16)
|
||||
ubatch 512 -> 1536 cost ROCm1 ~3.3 GiB against ROCm0's ~1.7, which left the
|
||||
draft-carrying card at 0.08 GiB free under a split that was balanced at the
|
||||
smaller batch. A split balanced at one batch is unbalanced at another: ~1.5
|
||||
GiB of load moves per `-ts` step at this size, so re-measure the balance after
|
||||
changing the batch (and remember 8192/2048 does not fit at 262144 at all --
|
||||
its 2020 MiB pp buffer fails to allocate). See
|
||||
[[frank-medium-llamacpp-dflash2]].
|
||||
- **ngram-mod prints no startup banner.** The load log shows the drafter's
|
||||
`adding speculative implementation 'draft-dflash'` and nothing at all for the
|
||||
matcher, so a chain can look unengaged while it is working. Identify it
|
||||
|
||||
+49
-36
@@ -54,12 +54,14 @@
|
||||
#
|
||||
# SIZING, measured (rocm-smi, 2026-09-16). Weights split across the two cards;
|
||||
# the drafter AND its KV ride the second. The loader's compute buffers are part
|
||||
# of what is seen below -- they are allocated at load, not per request:
|
||||
# of what is seen below -- they are allocated at load, not per request, and they
|
||||
# scale with --ubatch-size:
|
||||
#
|
||||
# CTX_SIZE split GPU 0 GPU 1 headroom on the busier card
|
||||
# 131072 1,1 19.89 GB 24.30 GB 9.9 GB / 31 % (DFlash2)
|
||||
# 262144 1,1 24.72 GB 29.14 GB 5.07 GB / 15 % (DFlash2) <- before
|
||||
# 262144 38,27 28.52 GB 30.71 GB 3.50 GB / 10 % (DSpark) <- deployed
|
||||
# CTX split batch slots GPU 0 free GPU 1 free note
|
||||
# 262144 1,1 2048 1 8.64 GiB 4.43 GiB DFlash2, day's start
|
||||
# 262144 38,27 2048 1 5.49 GiB 3.41 GiB DSpark, 1 slot
|
||||
# 262144 38,27 6144 2 3.78 GiB 0.08 GiB 3x batch is ASYMMETRIC
|
||||
# 262144 40,25 6144 2 1.87 GiB 1.86 GiB <- deployed
|
||||
#
|
||||
# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
|
||||
# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
|
||||
@@ -67,25 +69,32 @@
|
||||
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
|
||||
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
|
||||
#
|
||||
# WHY THE SPLIT IS 38:27. The draft context is built with `n_ctx =
|
||||
# llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so the drafter's KV scales
|
||||
# with the target window, and DSpark -- unlike DFlash2, which declared
|
||||
# sliding_window 2048 and cost ~40 MiB -- caches all of it: 5 layers x 2 x 8 kv
|
||||
# heads x 128 dim x 2 B = 20 KiB per position, i.e. 5.0 GiB at 262144. At a
|
||||
# 1:1 split that put ROCm1 at 33.9 GB of 34.2 GB and the draft's first graph
|
||||
# capture died with `ROCm error: out of memory` (in
|
||||
# common_speculative_impl_draft_dflash::process). Handing three target layers
|
||||
# to ROCm0 buys back ~3 GiB of headroom on both cards at the full window.
|
||||
# WHY THE SPLIT IS 40:25, AND WHY IT DEPENDS ON THE BATCH. Two effects pull in
|
||||
# opposite directions:
|
||||
# * THE DRAFT KEEPS ITS OWN FULL-CONTEXT KV ON ROCm1. The draft context is
|
||||
# built with `n_ctx = llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so
|
||||
# DSpark -- which, unlike DFlash2 (`sliding_window` 2048, ~40 MiB), declares
|
||||
# no window -- caches 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per
|
||||
# position: 5.0 GiB at 262144. At a 1:1 split that put ROCm1 at 33.9 GB of
|
||||
# 34.2 GB and the draft's first graph capture died with `ROCm error: out of
|
||||
# memory` (common_speculative_impl_draft_dflash::process). Hence 38:27.
|
||||
# * THE BIGGER BATCH COSTS ROCm1 ABOUT TWICE WHAT IT COSTS ROCm0 (~3.3 vs
|
||||
# ~1.7 GiB going 2048/512 -> 6144/1536 at 262144): ROCm1 holds the output
|
||||
# layers AND the drafter's graphs. At 38:27 that left ROCm1 at 0.08 GiB
|
||||
# free against ROCm0's 3.78 -- the reason for the rebalance.
|
||||
# One -ts step shifts ~1.5 GiB of load between the cards at this batch, so the
|
||||
# balance is found by measurement, not arithmetic. 40:25 is the measured
|
||||
# balance (a 0.05 GiB gap) and it is also what MAXIMISES the context: with the
|
||||
# floor at ~1.5 GiB per card, an unbalanced split would have to buy its
|
||||
# headroom out of CTX_SIZE.
|
||||
#
|
||||
# EQUALITY WAS TRIED AND REJECTED. 39:26 levels the cards more closely (3.86 /
|
||||
# 4.69 GiB free -- an 0.83 GiB gap, against 2.08 GiB here), but that server
|
||||
# died on its eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults
|
||||
# and the engine surfaced `ROCm error: an illegal memory access was
|
||||
# encountered` -> ggml_abort, i.e. a fault in the cross-card boundary copy,
|
||||
# which is this layer split's hot path. 38:27 has since served the whole e2e
|
||||
# gate plus a burst of varied requests with none. Do not chase an exact
|
||||
# balance: take the skew that fits and leave the drafter's card on the safe
|
||||
# side of the gap.
|
||||
# A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth
|
||||
# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
|
||||
# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
|
||||
# i.e. a fault in the cross-card boundary copy, this layer split's hot path. It
|
||||
# has not reproduced (38:27 served the whole e2e gate and a burst of requests;
|
||||
# 40:25 has served this file's verification battery), but the signature is
|
||||
# worth recognising -- it is NOT an OOM, and no amount of headroom prevents it.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
||||
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
||||
@@ -112,27 +121,31 @@ HIP_VISIBLE_DEVICES=0,1
|
||||
DEVICES=ROCm0,ROCm1
|
||||
SPLIT_MODE=layer
|
||||
|
||||
# Skewed 38:27 rather than 1:1: ROCm1 also carries the drafter and the whole of
|
||||
# its KV, so it must hold fewer target layers for the cards to end level -- at
|
||||
# 1:1 the drafter's card would sit ~9 GiB above the other. This is the measured
|
||||
# point that keeps BOTH cards above a 3 GiB floor at the full window; a tighter
|
||||
# balance was tried and rejected (see SIZING above).
|
||||
TENSOR_SPLIT=38,27
|
||||
# 40:25 -- the measured balance at the deployed batch, and the point that
|
||||
# maximises CTX_SIZE: ROCm1 carries the drafter, its full-context KV and the
|
||||
# output layers, so it takes fewer target layers. See SIZING above; one -ts
|
||||
# step is worth ~1.5 GiB at this batch, so do not move it without re-measuring.
|
||||
TENSOR_SPLIT=40,25
|
||||
|
||||
NGL=999
|
||||
# Auto-fit has no business resizing a context we set explicitly.
|
||||
FIT=off
|
||||
FLASH_ATTN=on
|
||||
|
||||
# One slot at the model's full configured window (n_ctx_train 262144), which was
|
||||
# measured to fit before it was deployed: see SIZING above. --parallel DIVIDES
|
||||
# --ctx-size between slots, so one slot carries the whole window and a second
|
||||
# concurrent request queues behind it.
|
||||
# The model's full window (n_ctx_train 262144) divided over TWO slots, so each
|
||||
# conversation gets 131072 -- verified at load as `n_slots = 2, n_ctx_slot =
|
||||
# 131072`. --parallel DIVIDES --ctx-size, so more slots would shrink every
|
||||
# conversation rather than grow the cache.
|
||||
CTX_SIZE=262144
|
||||
PARALLEL=1
|
||||
PARALLEL=2
|
||||
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=512
|
||||
# 3x the build's defaults (2048/512). --ubatch-size is the physical batch the
|
||||
# graph is built at and therefore what the compute buffers cost: at 262144 this
|
||||
# setting needs ~1.7 GiB more on ROCm0 and ~3.3 GiB on ROCm1 than 2048/512 does
|
||||
# -- which is what forced the rebalance above. 8192/2048 does not fit at all
|
||||
# here: its 2020 MiB pp buffer fails to allocate on ROCm1.
|
||||
BATCH_SIZE=6144
|
||||
UBATCH_SIZE=1536
|
||||
|
||||
# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
|
||||
NUM_THREADS=8
|
||||
|
||||
@@ -47,7 +47,7 @@ DRAFT_BYTES=1359284737 # DSpark Q8_0 drafter
|
||||
@test "both R9700s serve ONE model, layer-split inside one process" {
|
||||
grep -q '^SPLIT_MODE=layer$' "$ENV_FILE"
|
||||
grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE"
|
||||
grep -q '^TENSOR_SPLIT=38,27$' "$ENV_FILE"
|
||||
grep -q '^TENSOR_SPLIT=40,25$' "$ENV_FILE"
|
||||
grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE"
|
||||
grep -q -- '--device "$DEVICES"' "$SERVE"
|
||||
grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE"
|
||||
@@ -129,37 +129,57 @@ print('draft-model wiring ok')
|
||||
grep -q 'verifying sizes' "$FETCH"
|
||||
}
|
||||
|
||||
@test "one slot's KV still fits a card beside the weights that card holds" {
|
||||
local ctx parallel cache_k cache_v split_hi split_lo
|
||||
@test "both cards stay balanced and keep the operator's 1.5 GiB floor" {
|
||||
local ctx parallel cache_k cache_v split_hi split_lo batch ubatch
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")"
|
||||
batch="$(sed -nE 's/^BATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
ubatch="$(sed -nE 's/^UBATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ]
|
||||
[ -n "$split_hi" ] && [ -n "$split_lo" ]
|
||||
[ -n "$split_hi" ] && [ -n "$split_lo" ] && [ -n "$batch" ] && [ -n "$ubatch" ]
|
||||
[ "$parallel" -ge 1 ]
|
||||
[ "$batch" -ge "$ubatch" ]
|
||||
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
|
||||
# qwen35 spends KV on its 16 full-attention layers only: 2 (K,V) x 4 heads
|
||||
# x 256 dim x 2 B = 64 KiB per token at f16, carried by whichever card a
|
||||
# layer split puts those layers on. The busier card is the SECOND one: its
|
||||
# share of the weights and KV, plus the WHOLE drafter -- whose KV spans the
|
||||
# full context, because DSpark declares no sliding window (5 x 2 x 8 x 128
|
||||
# x 2 B = 20480 B per position) -- plus the loader's compute buffers.
|
||||
# Per card, calibrated against rocm-smi at the deployed point (40:25, ctx
|
||||
# 262144, batch 6144 x ubatch 1536, two slots, DSpark) where the two cards
|
||||
# measured 32.199e9 / 32.212e9 B used:
|
||||
#
|
||||
# This model is deliberately calibrated against rocm-smi: at 262144 / 38:27
|
||||
# it predicts 30.69e9 B against the 30.71e9 B measured on the busier card
|
||||
# (2026-09-16). Keep it that way -- a model that is pessimistic beyond a few
|
||||
# GiB would reject a context that fits.
|
||||
local per_card bufs
|
||||
bufs=$(( 3584 * 1024 * 1024 ))
|
||||
per_card=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
|
||||
+ DRAFT_BYTES \
|
||||
+ (ctx * 65536) * split_lo / (split_hi + split_lo) \
|
||||
+ ctx * 20480 \
|
||||
+ bufs ))
|
||||
[ "$per_card" -le "$CARD_BYTES" ]
|
||||
# card 0 = its share of the weights and of the KV (qwen35 spends KV on
|
||||
# its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim
|
||||
# x 2 B = 64 KiB per token at f16) plus ~1.4e6 B per ubatch unit
|
||||
# of compute buffers
|
||||
# card 1 = the same for its own share, plus the WHOLE drafter -- its
|
||||
# weights AND its full-context KV, because DSpark declares no
|
||||
# sliding window (5 x 2 x 8 x 128 x 2 B = 20480 B per position;
|
||||
# the draft context is sized from the target's:
|
||||
# `n_ctx = llama_n_ctx(ctx_tgt)`) -- and ~2.7e6 B per ubatch unit
|
||||
# on top of a ~2.3 GiB constant. Its buffers are the heavier ones
|
||||
# because it holds the output layers and the drafter's graphs.
|
||||
#
|
||||
# Both cards must clear the operator's floor, and a change to ctx, ubatch or
|
||||
# the split has to keep them from drifting apart -- an unbalanced split is
|
||||
# what cost ROCm1 3.7 GiB of headroom when the batch tripled (2026-09-16).
|
||||
local kv_pos=65536 draft_kv_pos=20480 floor slack
|
||||
local card_hi card_lo diff
|
||||
floor=$(( 1536 * 1024 * 1024 )) # 1.5 GiB per card
|
||||
slack=$(( 512 * 1024 * 1024 )) # measurement tolerance
|
||||
card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \
|
||||
+ kv_pos * ctx * split_hi / (split_hi + split_lo) \
|
||||
+ 1442 * 1024 * ubatch ))
|
||||
card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
|
||||
+ DRAFT_BYTES \
|
||||
+ kv_pos * ctx * split_lo / (split_hi + split_lo) \
|
||||
+ draft_kv_pos * ctx \
|
||||
+ 2353 * 1024 * 1024 \
|
||||
+ 2740 * 1024 * ubatch ))
|
||||
[ "$card_hi" -le $(( CARD_BYTES - floor )) ]
|
||||
[ "$card_lo" -le $(( CARD_BYTES - floor )) ]
|
||||
diff=$(( card_hi - card_lo )); [ "$diff" -ge 0 ] || diff=$(( -diff ))
|
||||
[ "$diff" -le "$slack" ]
|
||||
# ...and the window may not exceed what the checkpoint was trained for.
|
||||
[ "$ctx" -le 262144 ]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user