frank-medium: back to the DFlash2 drafter, and rebalance to 36:29
DSpark came off after a day. It was the better-looking file (1.27 GiB against 1.92) but it declares no sliding window, and the draft context is built with n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window (5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the draft-carrying card. The balance point moved with it, so it was re-measured at the deployed batch (6144 x 1536, 2 slots, 262144) rather than interpolated: 39,26 2.42 / 7.20 37,28 4.28 / 5.34 35,30 5.14 / 4.48 38,27 3.85 / 5.77 36,29 4.71 / 4.91 34,31 6.57 / 3.05 36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's 1.5 GiB floor on each card, so the full window stays. One -ts step is one layer, and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary, 1.43 GiB on a full-attention one -- which is why the table exists. Also in here: - the fetch script pointed at the wrong file (DSpark) AND declared the wrong size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made --verify-only fail on a payload that was in fact correct. Repointed at z-lab DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the Hub's LFS oids (host == Hub for both). - the seam's fit was recalibrated for a window-bounded draft KV and for the new point; it now passes 36:29 and fails every other split measured, including 1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which the comment says out loud. - the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll) are gone; the build's defaults apply.
This commit is contained in:
Frank
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the frank-medium payload for Dankserver (2x Radeon AI PRO R9700):
|
||||
# the target GGUF and the DSpark draft model, into /home/dank/models.
|
||||
# the target GGUF and the DFlash2 draft model, into /home/dank/models.
|
||||
#
|
||||
# bash scripts/download-frank-medium-gguf.sh
|
||||
# bash scripts/download-frank-medium-gguf.sh --verify-only
|
||||
@@ -17,10 +17,10 @@
|
||||
# own quant of the same size, on Unsloth's importance matrix). The file
|
||||
# carries the model's MTP layer (`blk.64.nextn.*`), so llama.cpp can also
|
||||
# draft from the model itself with no second copy in VRAM.
|
||||
# * Draft: magnitudedev/Qwen3.8-27B-DSpark-GGUF at Q8_0 -- the
|
||||
# block-diffusion drafter with the Markov head (5 layers, hidden 5120,
|
||||
# block_size 7). Q8_0 is 1.27 GiB against BF16's ~2.5 GiB and the
|
||||
# drafter's job is throughput, not output quality (lossless verification).
|
||||
# * Draft: z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0 -- the block-diffusion
|
||||
# drafter (5 layers, hidden 5120, block_size 8, a 2048-token sliding
|
||||
# window). Q8_0 is 1.92 GiB against BF16's 3.6 GiB and the drafter's job is
|
||||
# throughput, not output quality (the target verifies every drafted token).
|
||||
set -euo pipefail
|
||||
|
||||
MODEL_REPO="ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF"
|
||||
@@ -28,10 +28,10 @@ MODEL_DIR="/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF"
|
||||
MODEL_FILE="Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
|
||||
MODEL_SIZE=31457990112
|
||||
|
||||
DRAFT_REPO="magnitudedev/Qwen3.8-27B-DSpark-GGUF"
|
||||
DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DSpark-GGUF"
|
||||
DRAFT_FILE="Qwen3.8-27B-DSpark-Q8_0.gguf"
|
||||
DRAFT_SIZE=1359284737
|
||||
DRAFT_REPO="z-lab/Qwen3.8-27B-DFlash2-GGUF"
|
||||
DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DFlash2-GGUF"
|
||||
DRAFT_FILE="Qwen3.8-27B-DFlash2-Q8_0.gguf"
|
||||
DRAFT_SIZE=2056414816
|
||||
|
||||
mode="download"
|
||||
[[ "${1:-}" == "--verify-only" ]] && mode="verify"
|
||||
|
||||
+54
-43
@@ -1,5 +1,5 @@
|
||||
# Tunables for frank-medium (ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP
|
||||
# UD-Q8_K_XL + magnitudedev/Qwen3.8-27B-DSpark-GGUF Q8_0 drafter), mainline llama.cpp
|
||||
# UD-Q8_K_XL + z-lab/Qwen3.8-27B-DFlash2-GGUF Q8_0 drafter), mainline llama.cpp
|
||||
# on Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this
|
||||
# is the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
|
||||
# routing proxy are retired.
|
||||
@@ -39,11 +39,11 @@
|
||||
# carries the model's MTP head (blk.64.nextn.*), so a file-level switch to
|
||||
# --spec-type draft-mtp is available without a second copy in VRAM.
|
||||
#
|
||||
# Drafter: magnitudedev/Qwen3.8-27B-DSpark-GGUF @ Q8_0, 1.27 GiB: 5 layers,
|
||||
# hidden 5120, block-diffusion proposal with the Markov head (dspark).
|
||||
# Decoding stays lossless -- the target verifies every drafted token. Its KV
|
||||
# spans the whole context (see SIZING): unlike DFlash2 it declares no sliding
|
||||
# window, and that is what the tensor split below pays for.
|
||||
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q8_0, 1.92 GiB: 5 layers, hidden
|
||||
# 5120, a block-diffusion proposal (block_size 8) under a 2048-token sliding
|
||||
# window. Decoding stays lossless -- the target verifies every drafted token.
|
||||
# The window is what makes it cheap HERE: its KV is bounded at 2048 positions
|
||||
# (~40 MiB) instead of spanning the context (see SIZING).
|
||||
#
|
||||
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
|
||||
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
|
||||
@@ -57,11 +57,21 @@
|
||||
# of what is seen below -- they are allocated at load, not per request, and they
|
||||
# scale with --ubatch-size:
|
||||
#
|
||||
# CTX split batch slots GPU 0 free GPU 1 free note
|
||||
# 262144 1,1 2048 1 8.64 GiB 4.43 GiB DFlash2, day's start
|
||||
# 262144 38,27 2048 1 5.49 GiB 3.41 GiB DSpark, 1 slot
|
||||
# 262144 38,27 6144 2 3.78 GiB 0.08 GiB 3x batch is ASYMMETRIC
|
||||
# 262144 40,25 6144 2 1.87 GiB 1.86 GiB <- deployed
|
||||
# CTX split batch slots drafter GPU 0 free GPU 1 free note
|
||||
# 262144 1,1 2048 1 DFlash2 8.64 GiB 4.43 GiB day's start
|
||||
# 262144 39,26 6144 2 DFlash2 2.42 GiB 7.20 GiB
|
||||
# 262144 38,27 6144 2 DFlash2 3.85 GiB 5.77 GiB
|
||||
# 262144 37,28 6144 2 DFlash2 4.28 GiB 5.34 GiB
|
||||
# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB <- deployed
|
||||
# 262144 35,30 6144 2 DFlash2 5.14 GiB 4.48 GiB
|
||||
# 262144 40,25 6144 2 DSpark 1.87 GiB 1.86 GiB (retired)
|
||||
#
|
||||
# ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE. This is a hybrid
|
||||
# stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a
|
||||
# DeltaNet layer shifts ~0.43 GiB while an attention layer shifts ~1.43 GiB
|
||||
# (0.45 of weights plus ~1.0 GiB of its own f16 KV at 262144). Both step sizes
|
||||
# were measured, which is why the balance is found by measurement rather than by
|
||||
# interpolating between splits.
|
||||
#
|
||||
# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
|
||||
# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
|
||||
@@ -69,32 +79,30 @@
|
||||
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
|
||||
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
|
||||
#
|
||||
# WHY THE SPLIT IS 40:25, AND WHY IT DEPENDS ON THE BATCH. Two effects pull in
|
||||
# opposite directions:
|
||||
# * THE DRAFT KEEPS ITS OWN FULL-CONTEXT KV ON ROCm1. The draft context is
|
||||
# built with `n_ctx = llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so
|
||||
# DSpark -- which, unlike DFlash2 (`sliding_window` 2048, ~40 MiB), declares
|
||||
# no window -- caches 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per
|
||||
# position: 5.0 GiB at 262144. At a 1:1 split that put ROCm1 at 33.9 GB of
|
||||
# 34.2 GB and the draft's first graph capture died with `ROCm error: out of
|
||||
# memory` (common_speculative_impl_draft_dflash::process). Hence 38:27.
|
||||
# WHY THE SPLIT IS 36:29. Two effects both pull the balance BELOW an even
|
||||
# split:
|
||||
# * THE DRAFTER AND ITS OUTPUT LAYERS RIDE ROCm1. It carries the drafter's
|
||||
# weights, the draft graph and the last target layers, so it must be given
|
||||
# fewer.
|
||||
# * THE BIGGER BATCH COSTS ROCm1 ABOUT TWICE WHAT IT COSTS ROCm0 (~3.3 vs
|
||||
# ~1.7 GiB going 2048/512 -> 6144/1536 at 262144): ROCm1 holds the output
|
||||
# layers AND the drafter's graphs. At 38:27 that left ROCm1 at 0.08 GiB
|
||||
# free against ROCm0's 3.78 -- the reason for the rebalance.
|
||||
# One -ts step shifts ~1.5 GiB of load between the cards at this batch, so the
|
||||
# balance is found by measurement, not arithmetic. 40:25 is the measured
|
||||
# balance (a 0.05 GiB gap) and it is also what MAXIMISES the context: with the
|
||||
# floor at ~1.5 GiB per card, an unbalanced split would have to buy its
|
||||
# headroom out of CTX_SIZE.
|
||||
# layers AND the drafter's graphs. (Under the DSpark drafter that asymmetry
|
||||
# dominated the picture, because DSpark also caches the WHOLE context; its
|
||||
# KV alone was 5.0 GiB. DFlash2's window caps its KV at ~40 MiB, which
|
||||
# freed ~4.3 GiB on ROCm1 and moved the balance four layers.)
|
||||
# The layer-size granularity pushes back from the other side: 36:29 is the
|
||||
# measured minimum (0.20 GiB gap, against 0.65 at 35:30 and 1.06 at 37:28).
|
||||
# It also leaves the MOST ROOM -- this configuration clears the operator's
|
||||
# 1.5 GiB-per-card floor by 3.2 GiB, so no headroom has to be bought out of
|
||||
# CTX_SIZE and the full window stays.
|
||||
#
|
||||
# A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth
|
||||
# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
|
||||
# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
|
||||
# i.e. a fault in the cross-card boundary copy, this layer split's hot path. It
|
||||
# has not reproduced (38:27 served the whole e2e gate and a burst of requests;
|
||||
# 40:25 has served this file's verification battery), but the signature is
|
||||
# worth recognising -- it is NOT an OOM, and no amount of headroom prevents it.
|
||||
# has not reproduced (36:29 has served this file's verification battery, and the
|
||||
# e2e gate), but the signature is worth recognising -- it is NOT an OOM, and no
|
||||
# amount of headroom prevents it.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
||||
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
||||
@@ -106,8 +114,8 @@ LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama
|
||||
# Payload, fetched with scripts/download-frank-medium-gguf.sh.
|
||||
MODEL_DIR=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
||||
MODEL_FILE=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
||||
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DSpark-GGUF
|
||||
DRAFT_MODEL_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf
|
||||
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF
|
||||
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf
|
||||
|
||||
# Fleet identity. 7770 is the fleet's dankserver endpoint; the alias is the id
|
||||
# at /v1/models and what every client asks for.
|
||||
@@ -121,11 +129,12 @@ HIP_VISIBLE_DEVICES=0,1
|
||||
DEVICES=ROCm0,ROCm1
|
||||
SPLIT_MODE=layer
|
||||
|
||||
# 40:25 -- the measured balance at the deployed batch, and the point that
|
||||
# maximises CTX_SIZE: ROCm1 carries the drafter, its full-context KV and the
|
||||
# output layers, so it takes fewer target layers. See SIZING above; one -ts
|
||||
# step is worth ~1.5 GiB at this batch, so do not move it without re-measuring.
|
||||
TENSOR_SPLIT=40,25
|
||||
# 36:29 -- the measured balance at the deployed batch, and the split that leaves
|
||||
# the most room: ROCm1 carries the drafter, its KV and the output layers, so it
|
||||
# takes fewer target layers. See SIZING above for the measurement table; one
|
||||
# -ts step is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so
|
||||
# do not move it without re-measuring.
|
||||
TENSOR_SPLIT=36,29
|
||||
|
||||
NGL=999
|
||||
# Auto-fit has no business resizing a context we set explicitly.
|
||||
@@ -142,8 +151,10 @@ PARALLEL=2
|
||||
# 3x the build's defaults (2048/512). --ubatch-size is the physical batch the
|
||||
# graph is built at and therefore what the compute buffers cost: at 262144 this
|
||||
# setting needs ~1.7 GiB more on ROCm0 and ~3.3 GiB on ROCm1 than 2048/512 does
|
||||
# -- which is what forced the rebalance above. 8192/2048 does not fit at all
|
||||
# here: its 2020 MiB pp buffer fails to allocate on ROCm1.
|
||||
# -- which is what put the balance below an even split. (8192/2048 failed to
|
||||
# allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had
|
||||
# 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and with
|
||||
# DFlash2's ~4.9 GiB of headroom it has NOT been re-measured.)
|
||||
BATCH_SIZE=6144
|
||||
UBATCH_SIZE=1536
|
||||
|
||||
@@ -181,13 +192,13 @@ REASONING_BUDGET=-1
|
||||
CHAT_TEMPLATE_FILE=
|
||||
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
||||
|
||||
# Speculative decoding: the DSpark block-diffusion drafter PLUS the draftless
|
||||
# n-gram matcher. DSpark's GGUF carries block_size 7, so at most 6 draft
|
||||
# Speculative decoding: the DFlash2 block-diffusion drafter PLUS the draftless
|
||||
# n-gram matcher. DFlash2's GGUF carries block_size 8, so at most 7 draft
|
||||
# tokens per verification step (llama.cpp asserts n_max <= block_size - 1).
|
||||
# The drafter rides ROCm1, the card that ends up holding the target's last
|
||||
# layers.
|
||||
SPEC_TYPE=draft-dspark,ngram-mod
|
||||
SPEC_DRAFT_N_MAX=6
|
||||
SPEC_TYPE=draft-dflash,ngram-mod
|
||||
SPEC_DRAFT_N_MAX=7
|
||||
SPEC_DRAFT_NGL=999
|
||||
SPEC_DRAFT_DEVICE=ROCm1
|
||||
SPEC_DRAFT_TYPE_K=f16
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL as
|
||||
# frank-medium on Dankserver's two Radeon AI PRO R9700 (gfx1201), with the
|
||||
# magnitudedev DSpark block-diffusion drafter, using mainline llama.cpp.
|
||||
# z-lab DFlash2 block-diffusion drafter, using mainline llama.cpp.
|
||||
#
|
||||
# Runtime: llama.cpp master fb27a525 (2026-09-16), a gfx1201 HIP build that
|
||||
# bundles its own ROCm 7.14 runtime (RPATH $ORIGIN/../lib), produced by
|
||||
@@ -74,19 +74,17 @@ fi
|
||||
: "${CHAT_TEMPLATE_FILE:=}"
|
||||
: "${CHAT_TEMPLATE_KWARGS:={\"enable_thinking\":true,\"preserve_thinking\":true}}"
|
||||
|
||||
# Speculative decoding: draft-dflash (the DFlash2 drafter is a file, not a
|
||||
# flag) or draft-mtp (the target GGUF's own MTP head -- no draft file).
|
||||
# Speculative decoding: draft-dflash or draft-dspark (both are MODEL drafters --
|
||||
# a file, not a flag), or draft-mtp (the target GGUF's own MTP head, no draft
|
||||
# file). The drafter's own tuning knobs (n-min, p-min, p-split, poll) are left
|
||||
# at the build's defaults: they were only ever passed for dspark, and dspark is
|
||||
# not the deployed drafter.
|
||||
: "${SPEC_TYPE:=draft-dflash}"
|
||||
: "${SPEC_DRAFT_N_MAX:=7}"
|
||||
: "${SPEC_DRAFT_N_MIN:=0}"
|
||||
: "${SPEC_DRAFT_P_MIN:=0.0}"
|
||||
: "${SPEC_DRAFT_P_SPLIT:=0.10}"
|
||||
: "${SPEC_DRAFT_NGL:=999}"
|
||||
: "${SPEC_DRAFT_DEVICE:=}"
|
||||
: "${SPEC_DRAFT_TYPE_K:=f16}"
|
||||
: "${SPEC_DRAFT_TYPE_V:=f16}"
|
||||
: "${SPEC_DRAFT_POLL:=1}"
|
||||
: "${SPEC_DRAFT_POLL_BATCH:=1}"
|
||||
: "${SPEC_NGRAM_N_MATCH:=24}"
|
||||
: "${SPEC_NGRAM_N_MIN:=48}"
|
||||
: "${SPEC_NGRAM_N_MAX:=64}"
|
||||
@@ -129,9 +127,6 @@ if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
||||
--spec-draft-type-v "$SPEC_DRAFT_TYPE_V"
|
||||
)
|
||||
fi
|
||||
if [[ "$SPEC_TYPE" == *dspark* || ( "$SPEC_DRAFT_P_MIN" != "0.0" && "$SPEC_DRAFT_P_MIN" != "0" ) ]]; then
|
||||
SPEC_ARGS+=(--spec-draft-n-min "$SPEC_DRAFT_N_MIN" --spec-draft-p-min "$SPEC_DRAFT_P_MIN")
|
||||
fi
|
||||
# ngram-mod is draftless: it needs no draft context and its window is its
|
||||
# own (--spec-ngram-mod-*), so it can ride in the chain beside a model
|
||||
# drafter. It also costs no VRAM -- a CPU hash table.
|
||||
|
||||
Reference in New Issue
Block a user