frank-medium: swap the drafter to magnitudedev/Qwen3.8-27B-DSpark
DSpark (Q8_0, 1.27 GiB) replaces z-lab DFlash2 (1.92 GiB): same 5-layer block-diffusion shape, Markov head instead of the candidate selector, and block_size 7, so the draft depth drops 7 -> 6 (llama.cpp asserts n_max <= block_size - 1). The serve script's dspark branch never passed --spec-draft-model (only the dflash branch did), so dspark was not deployable as written; the model file gate now covers both types. Download script, env and the unit seam track the new file and its exact byte count.
This commit is contained in:
Frank
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the frank-medium payload for Dankserver (2x Radeon AI PRO R9700):
|
||||
# the target GGUF and the DFlash2 draft model, into /home/dank/models.
|
||||
# the target GGUF and the DSpark draft model, into /home/dank/models.
|
||||
#
|
||||
# bash scripts/download-frank-medium-gguf.sh
|
||||
# bash scripts/download-frank-medium-gguf.sh --verify-only
|
||||
@@ -17,10 +17,10 @@
|
||||
# own quant of the same size, on Unsloth's importance matrix). The file
|
||||
# carries the model's MTP layer (`blk.64.nextn.*`), so llama.cpp can also
|
||||
# draft from the model itself with no second copy in VRAM.
|
||||
# * Draft: z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0 -- the block-diffusion
|
||||
# drafter with the candidate selector (5 layers, hidden 5120). Q8_0 is
|
||||
# 2.0 GiB against BF16's 3.9 GiB and the drafter's job is throughput, not
|
||||
# output quality (verification is lossless either way).
|
||||
# * Draft: magnitudedev/Qwen3.8-27B-DSpark-GGUF at Q8_0 -- the
|
||||
# block-diffusion drafter with the Markov head (5 layers, hidden 5120,
|
||||
# block_size 7). Q8_0 is 1.27 GiB against BF16's ~2.5 GiB and the
|
||||
# drafter's job is throughput, not output quality (lossless verification).
|
||||
set -euo pipefail
|
||||
|
||||
MODEL_REPO="ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF"
|
||||
@@ -28,10 +28,10 @@ MODEL_DIR="/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF"
|
||||
MODEL_FILE="Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
|
||||
MODEL_SIZE=31457990112
|
||||
|
||||
DRAFT_REPO="z-lab/Qwen3.8-27B-DFlash2-GGUF"
|
||||
DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DFlash2-GGUF"
|
||||
DRAFT_FILE="Qwen3.8-27B-DFlash2-Q8_0.gguf"
|
||||
DRAFT_SIZE=2056414816
|
||||
DRAFT_REPO="magnitudedev/Qwen3.8-27B-DSpark-GGUF"
|
||||
DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DSpark-GGUF"
|
||||
DRAFT_FILE="Qwen3.8-27B-DSpark-Q8_0.gguf"
|
||||
DRAFT_SIZE=1359284737
|
||||
|
||||
mode="download"
|
||||
[[ "${1:-}" == "--verify-only" ]] && mode="verify"
|
||||
|
||||
+13
-15
@@ -1,5 +1,5 @@
|
||||
# Tunables for frank-medium (ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP
|
||||
# UD-Q8_K_XL + z-lab/Qwen3.8-27B-DFlash2-GGUF Q8_0 drafter), mainline llama.cpp
|
||||
# UD-Q8_K_XL + magnitudedev/Qwen3.8-27B-DSpark-GGUF Q8_0 drafter), mainline llama.cpp
|
||||
# on Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this
|
||||
# is the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
|
||||
# routing proxy are retired.
|
||||
@@ -39,9 +39,9 @@
|
||||
# carries the model's MTP head (blk.64.nextn.*), so a file-level switch to
|
||||
# --spec-type draft-mtp is available without a second copy in VRAM.
|
||||
#
|
||||
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q8_0, 1.92 GiB: 5 layers, hidden
|
||||
# 5120, block-diffusion proposal with the candidate selector. Decoding stays
|
||||
# lossless -- the target verifies every drafted token.
|
||||
# Drafter: magnitudedev/Qwen3.8-27B-DSpark-GGUF @ Q8_0, 1.27 GiB: 5 layers,
|
||||
# hidden 5120, block-diffusion proposal with the Markov head (dspark).
|
||||
# Decoding stays lossless -- the target verifies every drafted token.
|
||||
#
|
||||
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
|
||||
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
|
||||
@@ -74,8 +74,8 @@ LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama
|
||||
# Payload, fetched with scripts/download-frank-medium-gguf.sh.
|
||||
MODEL_DIR=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
||||
MODEL_FILE=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
||||
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF
|
||||
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf
|
||||
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DSpark-GGUF
|
||||
DRAFT_MODEL_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf
|
||||
|
||||
# Fleet identity. 7770 is the fleet's dankserver endpoint; the alias is the id
|
||||
# at /v1/models and what every client asks for.
|
||||
@@ -137,15 +137,13 @@ REASONING_BUDGET=-1
|
||||
CHAT_TEMPLATE_FILE=
|
||||
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
||||
|
||||
# Speculative decoding: the DFlash2 block-diffusion drafter PLUS the draftless
|
||||
# n-gram matcher. Measured against the drafter alone on 2026-09-16 (tables
|
||||
# below): the chain is byte-identical to DFlash2 by itself where n-grams cannot
|
||||
# match, and ~2.6x it where they can. DFlash2's GGUF carries block_size 8, so at
|
||||
# most 7 draft tokens per verification step (llama.cpp asserts
|
||||
# n_max <= block_size - 1); 7 is also what the drafter was published with. The
|
||||
# drafter rides ROCm1, the card that ends up holding the target's last layers.
|
||||
SPEC_TYPE=draft-dflash,ngram-mod
|
||||
SPEC_DRAFT_N_MAX=7
|
||||
# Speculative decoding: the DSpark block-diffusion drafter PLUS the draftless
|
||||
# n-gram matcher. DSpark's GGUF carries block_size 7, so at most 6 draft
|
||||
# tokens per verification step (llama.cpp asserts n_max <= block_size - 1).
|
||||
# The drafter rides ROCm1, the card that ends up holding the target's last
|
||||
# layers.
|
||||
SPEC_TYPE=draft-dspark,ngram-mod
|
||||
SPEC_DRAFT_N_MAX=6
|
||||
SPEC_DRAFT_NGL=999
|
||||
SPEC_DRAFT_DEVICE=ROCm1
|
||||
SPEC_DRAFT_TYPE_K=f16
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL as
|
||||
# frank-medium on Dankserver's two Radeon AI PRO R9700 (gfx1201), with the
|
||||
# z-lab DFlash2 block-diffusion drafter, using mainline llama.cpp.
|
||||
# magnitudedev DSpark block-diffusion drafter, using mainline llama.cpp.
|
||||
#
|
||||
# Runtime: llama.cpp master fb27a525 (2026-09-16), a gfx1201 HIP build that
|
||||
# bundles its own ROCm 7.14 runtime (RPATH $ORIGIN/../lib), produced by
|
||||
@@ -117,7 +117,7 @@ fi
|
||||
SPEC_ARGS=()
|
||||
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
||||
SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
|
||||
if [[ "$SPEC_TYPE" == *dflash* ]]; then
|
||||
if [[ "$SPEC_TYPE" == *dflash* || "$SPEC_TYPE" == *dspark* ]]; then
|
||||
if [[ ! -r "$DRAFT_MODEL" ]]; then
|
||||
echo "missing or unreadable draft model: $DRAFT_MODEL" >&2
|
||||
exit 1
|
||||
|
||||
Reference in New Issue
Block a user