frank-medium: back to the DFlash2 drafter, and rebalance to 36:29

DSpark came off after a day. It was the better-looking file (1.27 GiB against
1.92) but it declares no sliding window, and the draft context is built with
n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window
(5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds
its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the
draft-carrying card.

The balance point moved with it, so it was re-measured at the deployed batch
(6144 x 1536, 2 slots, 262144) rather than interpolated:

  39,26  2.42 / 7.20     37,28  4.28 / 5.34     35,30  5.14 / 4.48
  38,27  3.85 / 5.77     36,29  4.71 / 4.91     34,31  6.57 / 3.05

36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's
1.5 GiB floor on each card, so the full window stays. One -ts step is one layer,
and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary,
1.43 GiB on a full-attention one -- which is why the table exists.

Also in here:
- the fetch script pointed at the wrong file (DSpark) AND declared the wrong
  size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made
  --verify-only fail on a payload that was in fact correct. Repointed at z-lab
  DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the
  Hub's LFS oids (host == Hub for both).
- the seam's fit was recalibrated for a window-bounded draft KV and for the new
  point; it now passes 36:29 and fails every other split measured, including
  1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which
  the comment says out loud.
- the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll)
  are gone; the build's defaults apply.
This commit is contained in:
Frank committed 2026-09-16 20:59:20 -07:00
1 parent 718a283afc
commit 91546dbb95
5 files changed
+129 -102

No files matched your search

+6 -11
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
# Serve ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL as
# frank-medium on Dankserver's two Radeon AI PRO R9700 (gfx1201), with the
# magnitudedev DSpark block-diffusion drafter, using mainline llama.cpp.
# z-lab DFlash2 block-diffusion drafter, using mainline llama.cpp.
#
# Runtime: llama.cpp master fb27a525 (2026-09-16), a gfx1201 HIP build that
# bundles its own ROCm 7.14 runtime (RPATH $ORIGIN/../lib), produced by
@@ -74,19 +74,17 @@ fi
: "${CHAT_TEMPLATE_FILE:=}"
: "${CHAT_TEMPLATE_KWARGS:={\"enable_thinking\":true,\"preserve_thinking\":true}}"
# Speculative decoding: draft-dflash (the DFlash2 drafter is a file, not a
# flag) or draft-mtp (the target GGUF's own MTP head -- no draft file).
# Speculative decoding: draft-dflash or draft-dspark (both are MODEL drafters --
# a file, not a flag), or draft-mtp (the target GGUF's own MTP head, no draft
# file). The drafter's own tuning knobs (n-min, p-min, p-split, poll) are left
# at the build's defaults: they were only ever passed for dspark, and dspark is
# not the deployed drafter.
: "${SPEC_TYPE:=draft-dflash}"
: "${SPEC_DRAFT_N_MAX:=7}"
: "${SPEC_DRAFT_N_MIN:=0}"
: "${SPEC_DRAFT_P_MIN:=0.0}"
: "${SPEC_DRAFT_P_SPLIT:=0.10}"
: "${SPEC_DRAFT_NGL:=999}"
: "${SPEC_DRAFT_DEVICE:=}"
: "${SPEC_DRAFT_TYPE_K:=f16}"
: "${SPEC_DRAFT_TYPE_V:=f16}"
: "${SPEC_DRAFT_POLL:=1}"
: "${SPEC_DRAFT_POLL_BATCH:=1}"
: "${SPEC_NGRAM_N_MATCH:=24}"
: "${SPEC_NGRAM_N_MIN:=48}"
: "${SPEC_NGRAM_N_MAX:=64}"
@@ -129,9 +127,6 @@ if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
--spec-draft-type-v "$SPEC_DRAFT_TYPE_V"
)
fi
if [[ "$SPEC_TYPE" == *dspark* || ( "$SPEC_DRAFT_P_MIN" != "0.0" && "$SPEC_DRAFT_P_MIN" != "0" ) ]]; then
SPEC_ARGS+=(--spec-draft-n-min "$SPEC_DRAFT_N_MIN" --spec-draft-p-min "$SPEC_DRAFT_P_MIN")
fi
# ngram-mod is draftless: it needs no draft context and its window is its
# own (--spec-ngram-mod-*), so it can ride in the chain beside a model
# drafter. It also costs no VRAM -- a CPU hash table.