From b312042706a2c587a0ba6dd0b0b2442dc25c0f6c Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 20:17:27 -0700 Subject: [PATCH] frank-medium: 3x batch over 2 slots, and rebalance the split to pay for it The operator's revised spec: batch x3 (6144/1536), parallel 2, a 1.5 GiB per-card floor, and the context maximised inside it. Measured at 262144 with the old 38:27 split, 3x batch was brutally asymmetric: ROCm1 (output layers + the drafter's graphs) paid ~3.3 GiB where ROCm0 paid ~1.7, leaving the draft-carrying card at 0.08 GiB free against ROCm0's 3.78. One -ts step is worth ~1.5 GiB at this batch, so the balance is a measurement: 41:24 overshot (1.47 / 2.38), 40:25 landed at 1.92 / 1.96 GiB free. That balance is also what maximises the context -- an unbalanced split would have to buy its headroom out of CTX_SIZE, and CTX_SIZE is already at the model's trained window (262144), now split over two slots of 131072. Verified under two concurrent 1832-token prompts: 200/200, 1.87 / 1.86 GiB free, n_slots = 2, n_ctx_slot = 131072, tool calls intact, and DSpark drafting on BOTH slots (acceptance 0.38/0.22 and 0.52/0.43). 8192/2048 does not fit at this context at all: its 2020 MiB pp buffer fails to allocate on ROCm1, and 4096/1024 fits but leaves 1.11 GiB -- both recorded in the env's sizing table. The unit seam now models both cards (share + draft + ubatch-scaled buffers, calibrated to the rocm-smi figures to within 0.002%), asserts the 1.5 GiB floor on each, and asserts the two cards stay within 0.5 GiB of each other. Both assertions were confirmed to FAIL on a deliberately bad config (ubatch 8192, and a 1:1 split) rather than pass by construction. --- AGENTS.md | 15 ++++++- scripts/frank-medium.env | 85 +++++++++++++++++++++--------------- tests/frank_medium_unit.bats | 64 +++++++++++++++++---------- 3 files changed, 104 insertions(+), 60 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 88c1d6c..0a3d3d7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (38:27), 262144 ctx / 1 slot, text only | 7770 | +| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (40:25), 262144 ctx over 2 slots, batch 6144x1536, text only | 7770 | | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 | @@ -305,7 +305,9 @@ systemd unit `scripts/frank-medium.service`) serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL** (29.29 GiB) with the **magnitudedev DSpark** `Q8_0` drafter chained to the draftless **ngram-mod** matcher, in **one process across both R9700s**, -`--split-mode layer` at a **38:27 tensor split**, 262144 ctx / 1 slot, text only. +`--split-mode layer` at a **40:25 tensor split** (the balance point for the +deployed batch: the drafter rides ROCm1, so it takes fewer target layers), +262144 ctx over **2 slots** of 131072, batch 6144x1536, text only. **Layer split is why both cards now serve ONE model.** Tensor parallelism is impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism — @@ -509,6 +511,15 @@ linked runbooks. drafter's GGUF header before sizing any spec-decode deployment, and prefer a skewed `-ts` over a context cut when the full window matters. See [[frank-medium-llamacpp-dflash2]]. +- **The compute buffers of a bigger `--ubatch-size` land ~4x heavier on the card + that holds the output layers and the drafter.** On frank-medium (2026-09-16) + ubatch 512 -> 1536 cost ROCm1 ~3.3 GiB against ROCm0's ~1.7, which left the + draft-carrying card at 0.08 GiB free under a split that was balanced at the + smaller batch. A split balanced at one batch is unbalanced at another: ~1.5 + GiB of load moves per `-ts` step at this size, so re-measure the balance after + changing the batch (and remember 8192/2048 does not fit at 262144 at all -- + its 2020 MiB pp buffer fails to allocate). See + [[frank-medium-llamacpp-dflash2]]. - **ngram-mod prints no startup banner.** The load log shows the drafter's `adding speculative implementation 'draft-dflash'` and nothing at all for the matcher, so a chain can look unengaged while it is working. Identify it diff --git a/scripts/frank-medium.env b/scripts/frank-medium.env index a041c4a..f0cd8d5 100644 --- a/scripts/frank-medium.env +++ b/scripts/frank-medium.env @@ -54,12 +54,14 @@ # # SIZING, measured (rocm-smi, 2026-09-16). Weights split across the two cards; # the drafter AND its KV ride the second. The loader's compute buffers are part -# of what is seen below -- they are allocated at load, not per request: +# of what is seen below -- they are allocated at load, not per request, and they +# scale with --ubatch-size: # -# CTX_SIZE split GPU 0 GPU 1 headroom on the busier card -# 131072 1,1 19.89 GB 24.30 GB 9.9 GB / 31 % (DFlash2) -# 262144 1,1 24.72 GB 29.14 GB 5.07 GB / 15 % (DFlash2) <- before -# 262144 38,27 28.52 GB 30.71 GB 3.50 GB / 10 % (DSpark) <- deployed +# CTX split batch slots GPU 0 free GPU 1 free note +# 262144 1,1 2048 1 8.64 GiB 4.43 GiB DFlash2, day's start +# 262144 38,27 2048 1 5.49 GiB 3.41 GiB DSpark, 1 slot +# 262144 38,27 6144 2 3.78 GiB 0.08 GiB 3x batch is ASYMMETRIC +# 262144 40,25 6144 2 1.87 GiB 1.86 GiB <- deployed # # 262144 is the window the model ships (n_ctx_train) and it fits with room, so # the fleet gets the whole thing; the arithmetic that used to cap this endpoint @@ -67,25 +69,32 @@ # adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers # rather than assuming, and prefer CTX_SIZE over --gpu-layers. # -# WHY THE SPLIT IS 38:27. The draft context is built with `n_ctx = -# llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so the drafter's KV scales -# with the target window, and DSpark -- unlike DFlash2, which declared -# sliding_window 2048 and cost ~40 MiB -- caches all of it: 5 layers x 2 x 8 kv -# heads x 128 dim x 2 B = 20 KiB per position, i.e. 5.0 GiB at 262144. At a -# 1:1 split that put ROCm1 at 33.9 GB of 34.2 GB and the draft's first graph -# capture died with `ROCm error: out of memory` (in -# common_speculative_impl_draft_dflash::process). Handing three target layers -# to ROCm0 buys back ~3 GiB of headroom on both cards at the full window. +# WHY THE SPLIT IS 40:25, AND WHY IT DEPENDS ON THE BATCH. Two effects pull in +# opposite directions: +# * THE DRAFT KEEPS ITS OWN FULL-CONTEXT KV ON ROCm1. The draft context is +# built with `n_ctx = llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so +# DSpark -- which, unlike DFlash2 (`sliding_window` 2048, ~40 MiB), declares +# no window -- caches 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per +# position: 5.0 GiB at 262144. At a 1:1 split that put ROCm1 at 33.9 GB of +# 34.2 GB and the draft's first graph capture died with `ROCm error: out of +# memory` (common_speculative_impl_draft_dflash::process). Hence 38:27. +# * THE BIGGER BATCH COSTS ROCm1 ABOUT TWICE WHAT IT COSTS ROCm0 (~3.3 vs +# ~1.7 GiB going 2048/512 -> 6144/1536 at 262144): ROCm1 holds the output +# layers AND the drafter's graphs. At 38:27 that left ROCm1 at 0.08 GiB +# free against ROCm0's 3.78 -- the reason for the rebalance. +# One -ts step shifts ~1.5 GiB of load between the cards at this batch, so the +# balance is found by measurement, not arithmetic. 40:25 is the measured +# balance (a 0.05 GiB gap) and it is also what MAXIMISES the context: with the +# floor at ~1.5 GiB per card, an unbalanced split would have to buy its +# headroom out of CTX_SIZE. # -# EQUALITY WAS TRIED AND REJECTED. 39:26 levels the cards more closely (3.86 / -# 4.69 GiB free -- an 0.83 GiB gap, against 2.08 GiB here), but that server -# died on its eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults -# and the engine surfaced `ROCm error: an illegal memory access was -# encountered` -> ggml_abort, i.e. a fault in the cross-card boundary copy, -# which is this layer split's hot path. 38:27 has since served the whole e2e -# gate plus a burst of varied requests with none. Do not chase an exact -# balance: take the skew that fits and leave the drafter's card on the safe -# side of the gap. +# A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth +# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine +# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort, +# i.e. a fault in the cross-card boundary copy, this layer split's hot path. It +# has not reproduced (38:27 served the whole e2e gate and a burst of requests; +# 40:25 has served this file's verification battery), but the signature is +# worth recognising -- it is NOT an OOM, and no amount of headroom prevents it. # # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads # those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp @@ -112,27 +121,31 @@ HIP_VISIBLE_DEVICES=0,1 DEVICES=ROCm0,ROCm1 SPLIT_MODE=layer -# Skewed 38:27 rather than 1:1: ROCm1 also carries the drafter and the whole of -# its KV, so it must hold fewer target layers for the cards to end level -- at -# 1:1 the drafter's card would sit ~9 GiB above the other. This is the measured -# point that keeps BOTH cards above a 3 GiB floor at the full window; a tighter -# balance was tried and rejected (see SIZING above). -TENSOR_SPLIT=38,27 +# 40:25 -- the measured balance at the deployed batch, and the point that +# maximises CTX_SIZE: ROCm1 carries the drafter, its full-context KV and the +# output layers, so it takes fewer target layers. See SIZING above; one -ts +# step is worth ~1.5 GiB at this batch, so do not move it without re-measuring. +TENSOR_SPLIT=40,25 NGL=999 # Auto-fit has no business resizing a context we set explicitly. FIT=off FLASH_ATTN=on -# One slot at the model's full configured window (n_ctx_train 262144), which was -# measured to fit before it was deployed: see SIZING above. --parallel DIVIDES -# --ctx-size between slots, so one slot carries the whole window and a second -# concurrent request queues behind it. +# The model's full window (n_ctx_train 262144) divided over TWO slots, so each +# conversation gets 131072 -- verified at load as `n_slots = 2, n_ctx_slot = +# 131072`. --parallel DIVIDES --ctx-size, so more slots would shrink every +# conversation rather than grow the cache. CTX_SIZE=262144 -PARALLEL=1 +PARALLEL=2 -BATCH_SIZE=2048 -UBATCH_SIZE=512 +# 3x the build's defaults (2048/512). --ubatch-size is the physical batch the +# graph is built at and therefore what the compute buffers cost: at 262144 this +# setting needs ~1.7 GiB more on ROCm0 and ~3.3 GiB on ROCm1 than 2048/512 does +# -- which is what forced the rebalance above. 8192/2048 does not fit at all +# here: its 2020 MiB pp buffer fails to allocate on ROCm1. +BATCH_SIZE=6144 +UBATCH_SIZE=1536 # Threads cover tokenization, sampling and HTTP only: every layer is on a GPU. NUM_THREADS=8 diff --git a/tests/frank_medium_unit.bats b/tests/frank_medium_unit.bats index 7566337..4efec3f 100755 --- a/tests/frank_medium_unit.bats +++ b/tests/frank_medium_unit.bats @@ -47,7 +47,7 @@ DRAFT_BYTES=1359284737 # DSpark Q8_0 drafter @test "both R9700s serve ONE model, layer-split inside one process" { grep -q '^SPLIT_MODE=layer$' "$ENV_FILE" grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE" - grep -q '^TENSOR_SPLIT=38,27$' "$ENV_FILE" + grep -q '^TENSOR_SPLIT=40,25$' "$ENV_FILE" grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE" grep -q -- '--device "$DEVICES"' "$SERVE" grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE" @@ -129,37 +129,57 @@ print('draft-model wiring ok') grep -q 'verifying sizes' "$FETCH" } -@test "one slot's KV still fits a card beside the weights that card holds" { - local ctx parallel cache_k cache_v split_hi split_lo +@test "both cards stay balanced and keep the operator's 1.5 GiB floor" { + local ctx parallel cache_k cache_v split_hi split_lo batch ubatch ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")" cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")" cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")" split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")" split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")" + batch="$(sed -nE 's/^BATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" + ubatch="$(sed -nE 's/^UBATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" [ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] - [ -n "$split_hi" ] && [ -n "$split_lo" ] + [ -n "$split_hi" ] && [ -n "$split_lo" ] && [ -n "$batch" ] && [ -n "$ubatch" ] [ "$parallel" -ge 1 ] + [ "$batch" -ge "$ubatch" ] [ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ] - # qwen35 spends KV on its 16 full-attention layers only: 2 (K,V) x 4 heads - # x 256 dim x 2 B = 64 KiB per token at f16, carried by whichever card a - # layer split puts those layers on. The busier card is the SECOND one: its - # share of the weights and KV, plus the WHOLE drafter -- whose KV spans the - # full context, because DSpark declares no sliding window (5 x 2 x 8 x 128 - # x 2 B = 20480 B per position) -- plus the loader's compute buffers. + # Per card, calibrated against rocm-smi at the deployed point (40:25, ctx + # 262144, batch 6144 x ubatch 1536, two slots, DSpark) where the two cards + # measured 32.199e9 / 32.212e9 B used: # - # This model is deliberately calibrated against rocm-smi: at 262144 / 38:27 - # it predicts 30.69e9 B against the 30.71e9 B measured on the busier card - # (2026-09-16). Keep it that way -- a model that is pessimistic beyond a few - # GiB would reject a context that fits. - local per_card bufs - bufs=$(( 3584 * 1024 * 1024 )) - per_card=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \ - + DRAFT_BYTES \ - + (ctx * 65536) * split_lo / (split_hi + split_lo) \ - + ctx * 20480 \ - + bufs )) - [ "$per_card" -le "$CARD_BYTES" ] + # card 0 = its share of the weights and of the KV (qwen35 spends KV on + # its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim + # x 2 B = 64 KiB per token at f16) plus ~1.4e6 B per ubatch unit + # of compute buffers + # card 1 = the same for its own share, plus the WHOLE drafter -- its + # weights AND its full-context KV, because DSpark declares no + # sliding window (5 x 2 x 8 x 128 x 2 B = 20480 B per position; + # the draft context is sized from the target's: + # `n_ctx = llama_n_ctx(ctx_tgt)`) -- and ~2.7e6 B per ubatch unit + # on top of a ~2.3 GiB constant. Its buffers are the heavier ones + # because it holds the output layers and the drafter's graphs. + # + # Both cards must clear the operator's floor, and a change to ctx, ubatch or + # the split has to keep them from drifting apart -- an unbalanced split is + # what cost ROCm1 3.7 GiB of headroom when the batch tripled (2026-09-16). + local kv_pos=65536 draft_kv_pos=20480 floor slack + local card_hi card_lo diff + floor=$(( 1536 * 1024 * 1024 )) # 1.5 GiB per card + slack=$(( 512 * 1024 * 1024 )) # measurement tolerance + card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \ + + kv_pos * ctx * split_hi / (split_hi + split_lo) \ + + 1442 * 1024 * ubatch )) + card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \ + + DRAFT_BYTES \ + + kv_pos * ctx * split_lo / (split_hi + split_lo) \ + + draft_kv_pos * ctx \ + + 2353 * 1024 * 1024 \ + + 2740 * 1024 * ubatch )) + [ "$card_hi" -le $(( CARD_BYTES - floor )) ] + [ "$card_lo" -le $(( CARD_BYTES - floor )) ] + diff=$(( card_hi - card_lo )); [ "$diff" -ge 0 ] || diff=$(( -diff )) + [ "$diff" -le "$slack" ] # ...and the window may not exceed what the checkpoint was trained for. [ "$ctx" -le 262144 ] }