dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: pay for DSpark's draft KV with a 38:27 split

Browse Source 
DSpark declares no sliding window, so its draft context -- built with
n_ctx = llama_n_ctx(ctx_tgt) -- caches the whole target window: 5 layers x 2 x 8
kv heads x 128 dim x 2 B = 20 KiB/position, i.e. 5.0 GiB at 262144, where
DFlash2's sliding_window 2048 cost ~40 MiB. At a 1:1 split that put ROCm1 at
33.9 GB of 34.2 GB and the first draft graph capture died inside
common_speculative_impl_draft_dflash::process with "ROCm error: out of memory"
-- after the model had loaded and /health had answered, which is why the deploy
looked like a crash-loop rather than a sizing failure.

Handing three target layers to ROCm0 restores headroom on both cards at the full
window. Measured at 262144: 28.52 GB / 30.71 GB, i.e. 5.30 GiB and 3.26 GiB free
(was 2.7 GB free on the draft-carrying card with 1:1 + DFlash2). The unit seam
now models the busier card as a skewed share plus the drafter's full-context KV,
calibrated to the rocm-smi figure.

AGENTS.md gains the durable lesson: read sliding_window out of the drafter's
GGUF header before sizing a spec-decode deployment.
This commit is contained in:
Frank
2026-09-16 19:48:29 -07:00
parent 46b56ab5e8
commit b6d4914fcc
3 changed files with 64 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+19 -6
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process, 262144 ctx / 1 slot, text only | 7770 |
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (38:27), 262144 ctx / 1 slot, text only | 7770 |
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
@@ -298,14 +298,14 @@ len ≈ 3` (MTP engaged); an image request is answered from the projector while
the halogen side refuses images with a named reason; 22,362 MiB of the card's
24,576 MiB is in use; `n_slots = 1, n_ctx_slot = 131072`.
### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE)
### frank-medium — llama.cpp layer split, DSpark + ngram-mod (Dankserver, ACTIVE)
Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
systemd unit `scripts/frank-medium.service`) serving
`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the draftless
**ngram-mod** matcher, in **one process across both R9700s**, `--split-mode
layer`, 262144 ctx / 1 slot, text only.
(29.29 GiB) with the **magnitudedev DSpark** `Q8_0` drafter chained to the
draftless **ngram-mod** matcher, in **one process across both R9700s**,
`--split-mode layer` at a **38:27 tensor split**, 262144 ctx / 1 slot, text only.
**Layer split is why both cards now serve ONE model.** Tensor parallelism is
impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism —
@@ -496,6 +496,19 @@ linked runbooks.
doesn't contain MTP layers"*, because MTP builds its context against whatever
model the draft context holds. Two context-backed types cannot share one
process, and MTP's head only loads when MTP **is** the draft model.
- **A drafter's KV scales with the TARGET's context, and its window policy
decides how much that costs.** The draft context is built with
`cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is
no flag to cap it — so a drafter that declares NO sliding window caches the
whole window on whatever device `--spec-draft-device` names. Swapping DFlash2
(`dflash.attention.sliding_window = 2048`, ~40 MiB of draft KV) for DSpark
(no window; 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position,
so **5.0 GiB at 262144**) put ROCm1 at 33.9 GB of 34.2 GB and killed the
FIRST draft graph capture with `ROCm error: out of memory` — after the model
had loaded and `/health` had answered. Read `sliding_window` out of the
drafter's GGUF header before sizing any spec-decode deployment, and prefer a
skewed `-ts` over a context cut when the full window matters. See
[[frank-medium-llamacpp-dflash2]].
- **ngram-mod prints no startup banner.** The load log shows the drafter's
`adding speculative implementation 'draft-dflash'` and nothing at all for the
matcher, so a chain can look unengaged while it is working. Identify it
@@ -587,7 +600,7 @@ Active:
|---------|---------|
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
| frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` |
| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
| frank-medium (llama.cpp + DSpark, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
scripts/frank-medium.env
+24 -8
View File
@@ -41,7 +41,9 @@
#
# Drafter: magnitudedev/Qwen3.8-27B-DSpark-GGUF @ Q8_0, 1.27 GiB: 5 layers,
# hidden 5120, block-diffusion proposal with the Markov head (dspark).
# Decoding stays lossless -- the target verifies every drafted token.
# Decoding stays lossless -- the target verifies every drafted token. Its KV
# spans the whole context (see SIZING): unlike DFlash2 it declares no sliding
# window, and that is what the tensor split below pays for.
#
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
@@ -51,12 +53,13 @@
# state at any depth.
#
# SIZING, measured (rocm-smi, 2026-09-16). Weights split across the two cards;
# the drafter rides the second. The loader's compute buffers are part of what
# is seen below -- they are allocated at load, not per request:
# the drafter AND its KV ride the second. The loader's compute buffers are part
# of what is seen below -- they are allocated at load, not per request:
#
# CTX_SIZE GPU 0 GPU 1 headroom on the busier card
# 131072 19.89 GB 24.30 GB 9.9 GB / 31 %
# 262144 24.72 GB 29.14 GB 5.07 GB / 15 % <- deployed
# CTX_SIZE split GPU 0 GPU 1 headroom on the busier card
# 131072 1,1 19.89 GB 24.30 GB 9.9 GB / 31 % (DFlash2)
# 262144 1,1 24.72 GB 29.14 GB 5.07 GB / 15 % (DFlash2) <- before
# 262144 38,27 28.52 GB 30.71 GB 3.50 GB / 10 % (DSpark) <- deployed
#
# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
@@ -64,6 +67,16 @@
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
#
# WHY THE SPLIT IS 38:27. The draft context is built with `n_ctx =
# llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so the drafter's KV scales
# with the target window, and DSpark -- unlike DFlash2, which declared
# sliding_window 2048 and cost ~40 MiB -- caches all of it: 5 layers x 2 x 8 kv
# heads x 128 dim x 2 B = 20 KiB per position, i.e. 5.0 GiB at 262144. At a
# 1:1 split that put ROCm1 at 33.9 GB of 34.2 GB and the draft's first graph
# capture died with `ROCm error: out of memory` (in
# common_speculative_impl_draft_dflash::process). Handing three target layers
# to ROCm0 buys back ~3 GiB of headroom on both cards at the full window.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
# as flags only.
@@ -88,8 +101,11 @@ LISTEN_PORT=7770
HIP_VISIBLE_DEVICES=0,1
DEVICES=ROCm0,ROCm1
SPLIT_MODE=layer
# Empty = distribute layers by free VRAM (equal here, so 1,1 by hand).
TENSOR_SPLIT=1,1
# Skewed 38:27 rather than 1:1: ROCm1 also carries the drafter and the whole of
# its KV, so it must hold fewer target layers for the cards to end level. See
# SIZING above -- this is measured, not computed.
TENSOR_SPLIT=38,27
NGL=999
# Auto-fit has no business resizing a context we set explicitly.
tests/frank_medium_unit.bats
+21 -11
View File
@@ -47,7 +47,7 @@ DRAFT_BYTES=1359284737 # DSpark Q8_0 drafter
@test "both R9700s serve ONE model, layer-split inside one process" {
grep -q '^SPLIT_MODE=layer$' "$ENV_FILE"
grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE"
grep -q '^TENSOR_SPLIT=1,1$' "$ENV_FILE"
grep -q '^TENSOR_SPLIT=38,27$' "$ENV_FILE"
grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE"
grep -q -- '--device "$DEVICES"' "$SERVE"
grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE"
@@ -130,25 +130,35 @@ print('draft-model wiring ok')
}
@test "one slot's KV still fits a card beside the weights that card holds" {
local ctx parallel cache_k cache_v
local ctx parallel cache_k cache_v split_hi split_lo
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")"
split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")"
[ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ]
[ -n "$split_hi" ] && [ -n "$split_lo" ]
[ "$parallel" -ge 1 ]
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
# qwen35 spends KV on its 16 full-attention layers only: 2 (K,V) x 4 heads
# x 256 dim x 2 B = 64 KiB per token at f16. A layer split puts those
# layers (and their KV) on both cards, so the worst card carries half the
# weights, half the KV, the drafter and ~2 GiB of compute buffers.
# x 256 dim x 2 B = 64 KiB per token at f16, carried by whichever card a
# layer split puts those layers on. The busier card is the SECOND one: its
# share of the weights and KV, plus the WHOLE drafter -- whose KV spans the
# full context, because DSpark declares no sliding window (5 x 2 x 8 x 128
# x 2 B = 20480 B per position) -- plus the loader's compute buffers.
#
# This model is deliberately calibrated against rocm-smi: at 262144 it
# predicts 28.53e9 B against the 29.14e9 B measured on the busier card
# (2026-09-16), i.e. ~2 % optimistic. Keep it that way -- a model that is
# pessimistic beyond a few GiB would reject a context that fits.
local per_card
per_card=$(( MODEL_BYTES / 2 + DRAFT_BYTES + (ctx * 65536) / 2 + 2 * 1024 * 1024 * 1024 ))
# This model is deliberately calibrated against rocm-smi: at 262144 / 38:27
# it predicts 30.69e9 B against the 30.71e9 B measured on the busier card
# (2026-09-16). Keep it that way -- a model that is pessimistic beyond a few
# GiB would reject a context that fits.
local per_card bufs
bufs=$(( 3584 * 1024 * 1024 ))
per_card=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
+ DRAFT_BYTES \
+ (ctx * 65536) * split_lo / (split_hi + split_lo) \
+ ctx * 20480 \
+ bufs ))
[ "$per_card" -le "$CARD_BYTES" ]
# ...and the window may not exceed what the checkpoint was trained for.
[ "$ctx" -le 262144 ]