frank-medium: back to the DFlash2 drafter, and rebalance to 36:29
DSpark came off after a day. It was the better-looking file (1.27 GiB against 1.92) but it declares no sliding window, and the draft context is built with n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window (5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the draft-carrying card. The balance point moved with it, so it was re-measured at the deployed batch (6144 x 1536, 2 slots, 262144) rather than interpolated: 39,26 2.42 / 7.20 37,28 4.28 / 5.34 35,30 5.14 / 4.48 38,27 3.85 / 5.77 36,29 4.71 / 4.91 34,31 6.57 / 3.05 36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's 1.5 GiB floor on each card, so the full window stays. One -ts step is one layer, and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary, 1.43 GiB on a full-attention one -- which is why the table exists. Also in here: - the fetch script pointed at the wrong file (DSpark) AND declared the wrong size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made --verify-only fail on a payload that was in fact correct. Repointed at z-lab DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the Hub's LFS oids (host == Hub for both). - the seam's fit was recalibrated for a window-bounded draft KV and for the new point; it now passes 36:29 and fails every other split measured, including 1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which the comment says out loud. - the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll) are gone; the build's defaults apply.
This commit is contained in:
1 parent
718a283afc
commit
91546dbb95
5 files changed
+129
-102
No files matched your search
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL as
|
||||
# frank-medium on Dankserver's two Radeon AI PRO R9700 (gfx1201), with the
|
||||
# magnitudedev DSpark block-diffusion drafter, using mainline llama.cpp.
|
||||
# z-lab DFlash2 block-diffusion drafter, using mainline llama.cpp.
|
||||
#
|
||||
# Runtime: llama.cpp master fb27a525 (2026-09-16), a gfx1201 HIP build that
|
||||
# bundles its own ROCm 7.14 runtime (RPATH $ORIGIN/../lib), produced by
|
||||
@@ -74,19 +74,17 @@ fi
|
||||
: "${CHAT_TEMPLATE_FILE:=}"
|
||||
: "${CHAT_TEMPLATE_KWARGS:={\"enable_thinking\":true,\"preserve_thinking\":true}}"
|
||||
|
||||
# Speculative decoding: draft-dflash (the DFlash2 drafter is a file, not a
|
||||
# flag) or draft-mtp (the target GGUF's own MTP head -- no draft file).
|
||||
# Speculative decoding: draft-dflash or draft-dspark (both are MODEL drafters --
|
||||
# a file, not a flag), or draft-mtp (the target GGUF's own MTP head, no draft
|
||||
# file). The drafter's own tuning knobs (n-min, p-min, p-split, poll) are left
|
||||
# at the build's defaults: they were only ever passed for dspark, and dspark is
|
||||
# not the deployed drafter.
|
||||
: "${SPEC_TYPE:=draft-dflash}"
|
||||
: "${SPEC_DRAFT_N_MAX:=7}"
|
||||
: "${SPEC_DRAFT_N_MIN:=0}"
|
||||
: "${SPEC_DRAFT_P_MIN:=0.0}"
|
||||
: "${SPEC_DRAFT_P_SPLIT:=0.10}"
|
||||
: "${SPEC_DRAFT_NGL:=999}"
|
||||
: "${SPEC_DRAFT_DEVICE:=}"
|
||||
: "${SPEC_DRAFT_TYPE_K:=f16}"
|
||||
: "${SPEC_DRAFT_TYPE_V:=f16}"
|
||||
: "${SPEC_DRAFT_POLL:=1}"
|
||||
: "${SPEC_DRAFT_POLL_BATCH:=1}"
|
||||
: "${SPEC_NGRAM_N_MATCH:=24}"
|
||||
: "${SPEC_NGRAM_N_MIN:=48}"
|
||||
: "${SPEC_NGRAM_N_MAX:=64}"
|
||||
@@ -129,9 +127,6 @@ if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
||||
--spec-draft-type-v "$SPEC_DRAFT_TYPE_V"
|
||||
)
|
||||
fi
|
||||
if [[ "$SPEC_TYPE" == *dspark* || ( "$SPEC_DRAFT_P_MIN" != "0.0" && "$SPEC_DRAFT_P_MIN" != "0" ) ]]; then
|
||||
SPEC_ARGS+=(--spec-draft-n-min "$SPEC_DRAFT_N_MIN" --spec-draft-p-min "$SPEC_DRAFT_P_MIN")
|
||||
fi
|
||||
# ngram-mod is draftless: it needs no draft context and its window is its
|
||||
# own (--spec-ngram-mod-*), so it can ride in the chain beside a model
|
||||
# drafter. It also costs no VRAM -- a CPU hash table.
|
||||
|
||||
Reference in new issue
Block a user