dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: back to the DFlash2 drafter, and rebalance to 36:29

Browse Source 
DSpark came off after a day. It was the better-looking file (1.27 GiB against
1.92) but it declares no sliding window, and the draft context is built with
n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window
(5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds
its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the
draft-carrying card.

The balance point moved with it, so it was re-measured at the deployed batch
(6144 x 1536, 2 slots, 262144) rather than interpolated:

  39,26  2.42 / 7.20     37,28  4.28 / 5.34     35,30  5.14 / 4.48
  38,27  3.85 / 5.77     36,29  4.71 / 4.91     34,31  6.57 / 3.05

36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's
1.5 GiB floor on each card, so the full window stays. One -ts step is one layer,
and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary,
1.43 GiB on a full-attention one -- which is why the table exists.

Also in here:
- the fetch script pointed at the wrong file (DSpark) AND declared the wrong
  size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made
  --verify-only fail on a payload that was in fact correct. Repointed at z-lab
  DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the
  Hub's LFS oids (host == Hub for both).
- the seam's fit was recalibrated for a window-bounded draft KV and for the new
  point; it now passes 36:29 and fails every other split measured, including
  1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which
  the comment says out loud.
- the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll)
  are gone; the build's defaults apply.
This commit is contained in:
Frank
2026-09-16 20:59:20 -07:00
parent 718a283afc
commit 91546dbb95
5 changed files with 129 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_medium_unit.bats
+30 -23
View File
@@ -5,7 +5,7 @@
# day. What this seam pins is what the flags alone do not say: boot persistence
# for a NATIVE runtime (no generated quadlet is involved), that BOTH cards work
# on ONE model through a layer split with no collective in the path, that the
# DSpark drafter is wired with a depth its block size actually allows, that the
# DFlash2 drafter is wired with a depth its block size actually allows, that the
# runtime and payload are pinned rather than floated, and that one slot's KV
# still fits a 32 GiB card beside the weights it holds.
@@ -21,7 +21,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
# Measured constants (rocm-smi / HF blob sizes), used by the fit arithmetic.
CARD_BYTES=34208743424 # one R9700 as rocm-smi reports it
MODEL_BYTES=31457990112 # UD-Q8_K_XL target
DRAFT_BYTES=1359284737 # DSpark Q8_0 drafter
DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter
@test "frank-medium's unit, serve script, env and its build/fetch scripts exist" {
[ -f "$UNIT" ]
@@ -47,7 +47,7 @@ DRAFT_BYTES=1359284737 # DSpark Q8_0 drafter
@test "both R9700s serve ONE model, layer-split inside one process" {
grep -q '^SPLIT_MODE=layer$' "$ENV_FILE"
grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE"
grep -q '^TENSOR_SPLIT=40,25$' "$ENV_FILE"
grep -q '^TENSOR_SPLIT=36,29$' "$ENV_FILE"
grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE"
grep -q -- '--device "$DEVICES"' "$SERVE"
grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE"
@@ -65,8 +65,8 @@ DRAFT_BYTES=1359284737 # DSpark Q8_0 drafter
# The chain is the measured winner (see the runbook's A/B): free where the
# matcher cannot match, ~2.6x the drafter alone on prompt-echo traffic. A
# bare `draft-dflash` here would silently give that up.
grep -qE '^SPEC_TYPE=draft-dspark,ngram-mod$' "$ENV_FILE"
grep -q '^DRAFT_MODEL_FILE=Qwen3\.8-27B-DSpark-Q8_0\.gguf$' "$ENV_FILE"
grep -qE '^SPEC_TYPE=draft-dflash,ngram-mod$' "$ENV_FILE"
grep -q '^DRAFT_MODEL_FILE=Qwen3\.8-27B-DFlash2-Q8_0\.gguf$' "$ENV_FILE"
grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
grep -q -- '--spec-draft-model "$DRAFT_MODEL"' "$SERVE"
# ngram-mod's own window -- and only when the type asks for it, so a pure
@@ -85,11 +85,11 @@ end = next(i for i in range(guard + 1, len(lines))
if lines[i].strip() == 'fi' and len(lines[i]) - len(lines[i].lstrip()) == indent)
assert '--spec-ngram-mod-n-match' in chr(10).join(lines[guard:end]), 'ngram flags must sit in the ngram branch'
"
# DSpark drafts into a block of 7, so at most 6 draft tokens per
# DFlash2 drafts into a block of 8, so at most 7 draft tokens per
# verification step -- llama.cpp asserts n_max <= block_size - 1.
local n_max
n_max="$(sed -nE 's/^SPEC_DRAFT_N_MAX=([0-9]+)$/\1/p' "$ENV_FILE")"
[ -n "$n_max" ] && [ "$n_max" -ge 1 ] && [ "$n_max" -le 6 ]
[ -n "$n_max" ] && [ "$n_max" -ge 1 ] && [ "$n_max" -le 7 ]
}
@test "a draft MODEL is passed only for DFlash types, never for MTP" {
@@ -124,7 +124,7 @@ print('draft-model wiring ok')
@test "the payload is fetched resumably and verified by size" {
grep -q 'UD-Q8_K_XL\.gguf' "$FETCH"
grep -q 'Qwen3\.8-27B-DSpark-Q8_0\.gguf' "$FETCH"
grep -q 'Qwen3\.8-27B-DFlash2-Q8_0\.gguf' "$FETCH"
grep -q 'curl -fL -C -' "$FETCH"
grep -q 'verifying sizes' "$FETCH"
}
@@ -144,25 +144,32 @@ print('draft-model wiring ok')
[ "$parallel" -ge 1 ]
[ "$batch" -ge "$ubatch" ]
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
# Per card, calibrated against rocm-smi at the deployed point (40:25, ctx
# 262144, batch 6144 x ubatch 1536, two slots, DSpark) where the two cards
# measured 32.199e9 / 32.212e9 B used:
# Per card, calibrated against rocm-smi at the deployed point (36:29, ctx
# 262144, batch 6144 x ubatch 1536, two slots, DFlash2), where the two cards
# measured 4.71 / 4.91 GiB free. The fit lands within ~0.05 GiB of that.
#
# card 0 = its share of the weights and of the KV (qwen35 spends KV on
# its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim
# x 2 B = 64 KiB per token at f16) plus ~1.4e6 B per ubatch unit
# of compute buffers
# card 1 = the same for its own share, plus the WHOLE drafter -- its
# weights AND its full-context KV, because DSpark declares no
# sliding window (5 x 2 x 8 x 128 x 2 B = 20480 B per position;
# the draft context is sized from the target's:
# `n_ctx = llama_n_ctx(ctx_tgt)`) -- and ~2.7e6 B per ubatch unit
# on top of a ~2.3 GiB constant. Its buffers are the heavier ones
# because it holds the output layers and the drafter's graphs.
# card 1 = the same for its own share, plus the WHOLE drafter: its weights
# (2.06e9 B), its KV -- BOUNDED BY ITS SLIDING WINDOW, so
# 20480 B/pos x 2048 positions rather than the context -- and
# ~2.7e6 B per ubatch unit of buffers on top of a ~4.8 GiB
# constant. Its buffers are the heavier ones because it holds the
# output layers and the drafter's graphs.
#
# Both cards must clear the operator's floor, and a change to ctx, ubatch or
# the split has to keep them from drifting apart -- an unbalanced split is
# what cost ROCm1 3.7 GiB of headroom when the batch tripled (2026-09-16).
# The layer it charges for is the AVERAGE (0.70 GiB: 31.46e9 of weights plus
# 16 GiB of KV over 65 layers). This stack is a hybrid, so real steps vary
# around that -- 0.43 GiB for a DeltaNet layer, 1.43 GiB for a full-attention
# one -- which makes the drift check below CONSERVATIVE: it overstates the
# already trips it. That is the intent: moving the split means re-measuring,
# not re-arguing.
#
# The ubatch coefficients ARE fitted at 1536, so the floor check's "an
# 8192/2048 batch does not fit" is an EXTRAPOLATION, not a measurement --
# and it cannot see CONTIGUITY at all, which is the actual mechanism by which
# the DSpark-era 8192/2048 failed (a 2020 MiB pp buffer that found no room).
local kv_pos=65536 draft_kv_pos=20480 floor slack
local card_hi card_lo diff
floor=$(( 1536 * 1024 * 1024 )) # 1.5 GiB per card
@@ -173,8 +180,8 @@ print('draft-model wiring ok')
card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
+ DRAFT_BYTES \
+ kv_pos * ctx * split_lo / (split_hi + split_lo) \
+ draft_kv_pos * ctx \
+ 2353 * 1024 * 1024 \
+ draft_kv_pos * (ctx < 2048 ? ctx : 2048) \
+ 828697861 \
+ 2740 * 1024 * ubatch ))
[ "$card_hi" -le $(( CARD_BYTES - floor )) ]
[ "$card_lo" -le $(( CARD_BYTES - floor )) ]