frank-medium: back to the DFlash2 drafter, and rebalance to 36:29

DSpark came off after a day. It was the better-looking file (1.27 GiB against
1.92) but it declares no sliding window, and the draft context is built with
n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window
(5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds
its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the
draft-carrying card.

The balance point moved with it, so it was re-measured at the deployed batch
(6144 x 1536, 2 slots, 262144) rather than interpolated:

  39,26  2.42 / 7.20     37,28  4.28 / 5.34     35,30  5.14 / 4.48
  38,27  3.85 / 5.77     36,29  4.71 / 4.91     34,31  6.57 / 3.05

36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's
1.5 GiB floor on each card, so the full window stays. One -ts step is one layer,
and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary,
1.43 GiB on a full-attention one -- which is why the table exists.

Also in here:
- the fetch script pointed at the wrong file (DSpark) AND declared the wrong
  size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made
  --verify-only fail on a payload that was in fact correct. Repointed at z-lab
  DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the
  Hub's LFS oids (host == Hub for both).
- the seam's fit was recalibrated for a window-bounded draft KV and for the new
  point; it now passes 36:29 and fails every other split measured, including
  1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which
  the comment says out loud.
- the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll)
  are gone; the build's defaults apply.
This commit is contained in:
Frank committed 2026-09-16 20:59:20 -07:00
1 parent 718a283afc
commit 91546dbb95
5 files changed
+129 -102

No files matched your search

+9 -9
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# Download the frank-medium payload for Dankserver (2x Radeon AI PRO R9700):
# the target GGUF and the DSpark draft model, into /home/dank/models.
# the target GGUF and the DFlash2 draft model, into /home/dank/models.
#
# bash scripts/download-frank-medium-gguf.sh
# bash scripts/download-frank-medium-gguf.sh --verify-only
@@ -17,10 +17,10 @@
# own quant of the same size, on Unsloth's importance matrix). The file
# carries the model's MTP layer (`blk.64.nextn.*`), so llama.cpp can also
# draft from the model itself with no second copy in VRAM.
# * Draft: magnitudedev/Qwen3.8-27B-DSpark-GGUF at Q8_0 -- the
# block-diffusion drafter with the Markov head (5 layers, hidden 5120,
# block_size 7). Q8_0 is 1.27 GiB against BF16's ~2.5 GiB and the
# drafter's job is throughput, not output quality (lossless verification).
# * Draft: z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0 -- the block-diffusion
# drafter (5 layers, hidden 5120, block_size 8, a 2048-token sliding
# window). Q8_0 is 1.92 GiB against BF16's 3.6 GiB and the drafter's job is
# throughput, not output quality (the target verifies every drafted token).
set -euo pipefail
MODEL_REPO="ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF"
@@ -28,10 +28,10 @@ MODEL_DIR="/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF"
MODEL_FILE="Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
MODEL_SIZE=31457990112
DRAFT_REPO="magnitudedev/Qwen3.8-27B-DSpark-GGUF"
DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DSpark-GGUF"
DRAFT_FILE="Qwen3.8-27B-DSpark-Q8_0.gguf"
DRAFT_SIZE=1359284737
DRAFT_REPO="z-lab/Qwen3.8-27B-DFlash2-GGUF"
DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DFlash2-GGUF"
DRAFT_FILE="Qwen3.8-27B-DFlash2-Q8_0.gguf"
DRAFT_SIZE=2056414816
mode="download"
[[ "${1:-}" == "--verify-only" ]] && mode="verify"