From 91546dbb95f37b7109307b35724e34e70413876f Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 20:59:20 -0700 Subject: [PATCH] frank-medium: back to the DFlash2 drafter, and rebalance to 36:29 DSpark came off after a day. It was the better-looking file (1.27 GiB against 1.92) but it declares no sliding window, and the draft context is built with n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window (5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the draft-carrying card. The balance point moved with it, so it was re-measured at the deployed batch (6144 x 1536, 2 slots, 262144) rather than interpolated: 39,26 2.42 / 7.20 37,28 4.28 / 5.34 35,30 5.14 / 4.48 38,27 3.85 / 5.77 36,29 4.71 / 4.91 34,31 6.57 / 3.05 36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's 1.5 GiB floor on each card, so the full window stays. One -ts step is one layer, and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary, 1.43 GiB on a full-attention one -- which is why the table exists. Also in here: - the fetch script pointed at the wrong file (DSpark) AND declared the wrong size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made --verify-only fail on a payload that was in fact correct. Repointed at z-lab DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the Hub's LFS oids (host == Hub for both). - the seam's fit was recalibrated for a window-bounded draft KV and for the new point; it now passes 36:29 and fails every other split measured, including 1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which the comment says out loud. - the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll) are gone; the build's defaults apply. --- AGENTS.md | 46 ++++++++----- scripts/download-frank-medium-gguf.sh | 18 ++--- scripts/frank-medium.env | 97 +++++++++++++++------------ scripts/serve-frank-medium.sh | 17 ++--- tests/frank_medium_unit.bats | 53 ++++++++------- 5 files changed, 129 insertions(+), 102 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0a3d3d7..a9dbe8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (40:25), 262144 ctx over 2 slots, batch 6144x1536, text only | 7770 | +| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 262144 ctx over 2 slots, batch 6144x1536, text only | 7770 | | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 | @@ -298,16 +298,19 @@ len ≈ 3` (MTP engaged); an image request is answered from the projector while the halogen side refuses images with a named reason; 22,362 MiB of the card's 24,576 MiB is in use; `n_slots = 1, n_ctx_slot = 131072`. -### frank-medium — llama.cpp layer split, DSpark + ngram-mod (Dankserver, ACTIVE) +### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE) Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root systemd unit `scripts/frank-medium.service`) serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL** -(29.29 GiB) with the **magnitudedev DSpark** `Q8_0` drafter chained to the +(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the draftless **ngram-mod** matcher, in **one process across both R9700s**, -`--split-mode layer` at a **40:25 tensor split** (the balance point for the +`--split-mode layer` at a **36:29 tensor split** (the balance point for the deployed batch: the drafter rides ROCm1, so it takes fewer target layers), -262144 ctx over **2 slots** of 131072, batch 6144x1536, text only. +262144 ctx over **2 slots** of 131072, batch 6144x1536, text only. DSpark was +put up in the drafter's place the same day and reverted: it declares no sliding +window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four +layers of the split. **Layer split is why both cards now serve ONE model.** Tensor parallelism is impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism — @@ -509,17 +512,28 @@ linked runbooks. FIRST draft graph capture with `ROCm error: out of memory` — after the model had loaded and `/health` had answered. Read `sliding_window` out of the drafter's GGUF header before sizing any spec-decode deployment, and prefer a - skewed `-ts` over a context cut when the full window matters. See - [[frank-medium-llamacpp-dflash2]]. -- **The compute buffers of a bigger `--ubatch-size` land ~4x heavier on the card - that holds the output layers and the drafter.** On frank-medium (2026-09-16) - ubatch 512 -> 1536 cost ROCm1 ~3.3 GiB against ROCm0's ~1.7, which left the - draft-carrying card at 0.08 GiB free under a split that was balanced at the - smaller batch. A split balanced at one batch is unbalanced at another: ~1.5 - GiB of load moves per `-ts` step at this size, so re-measure the balance after - changing the batch (and remember 8192/2048 does not fit at 262144 at all -- - its 2020 MiB pp buffer fails to allocate). See + skewed `-ts` over a context cut when the full window matters. The pair was + swapped DFlash2 -> DSpark -> DFlash2 in one day (2026-09-16) and the round + trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 -> + 36:29), which is the practical size of the effect. See [[frank-medium-llamacpp-dflash2]]. +- **The compute buffers of a bigger `--ubatch-size` land about TWICE as heavy on + the card that holds the output layers and the drafter.** On frank-medium + (2026-09-16) ubatch 512 -> 1536 cost ROCm1 ~3.3 GiB against ROCm0's ~1.7, + which left the draft-carrying card at 0.08 GiB free under a split that was + balanced at the smaller batch. A split balanced at one batch is unbalanced at + another, so re-measure the balance after changing the batch. (8192/2048 failed + to allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 + had 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and + it has not been re-measured since.) +- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same + size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is + worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one + (0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144) — both + measured on frank-medium. Balance by measurement, never by interpolating + between two splits; an affine fit reproduces the deployed point and overstates + the gaps in between. `tests/frank_medium_unit.bats` encodes that fit as a + gate. See [[frank-medium-llamacpp-dflash2]]. - **ngram-mod prints no startup banner.** The load log shows the drafter's `adding speculative implementation 'draft-dflash'` and nothing at all for the matcher, so a chain can look unengaged while it is working. Identify it @@ -628,7 +642,7 @@ Active: |---------|---------| | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | | frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` | -| frank-medium (llama.cpp + DSpark, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` | +| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | diff --git a/scripts/download-frank-medium-gguf.sh b/scripts/download-frank-medium-gguf.sh index bbd85a3..90934a3 100755 --- a/scripts/download-frank-medium-gguf.sh +++ b/scripts/download-frank-medium-gguf.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Download the frank-medium payload for Dankserver (2x Radeon AI PRO R9700): -# the target GGUF and the DSpark draft model, into /home/dank/models. +# the target GGUF and the DFlash2 draft model, into /home/dank/models. # # bash scripts/download-frank-medium-gguf.sh # bash scripts/download-frank-medium-gguf.sh --verify-only @@ -17,10 +17,10 @@ # own quant of the same size, on Unsloth's importance matrix). The file # carries the model's MTP layer (`blk.64.nextn.*`), so llama.cpp can also # draft from the model itself with no second copy in VRAM. -# * Draft: magnitudedev/Qwen3.8-27B-DSpark-GGUF at Q8_0 -- the -# block-diffusion drafter with the Markov head (5 layers, hidden 5120, -# block_size 7). Q8_0 is 1.27 GiB against BF16's ~2.5 GiB and the -# drafter's job is throughput, not output quality (lossless verification). +# * Draft: z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0 -- the block-diffusion +# drafter (5 layers, hidden 5120, block_size 8, a 2048-token sliding +# window). Q8_0 is 1.92 GiB against BF16's 3.6 GiB and the drafter's job is +# throughput, not output quality (the target verifies every drafted token). set -euo pipefail MODEL_REPO="ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF" @@ -28,10 +28,10 @@ MODEL_DIR="/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF" MODEL_FILE="Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf" MODEL_SIZE=31457990112 -DRAFT_REPO="magnitudedev/Qwen3.8-27B-DSpark-GGUF" -DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DSpark-GGUF" -DRAFT_FILE="Qwen3.8-27B-DSpark-Q8_0.gguf" -DRAFT_SIZE=1359284737 +DRAFT_REPO="z-lab/Qwen3.8-27B-DFlash2-GGUF" +DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DFlash2-GGUF" +DRAFT_FILE="Qwen3.8-27B-DFlash2-Q8_0.gguf" +DRAFT_SIZE=2056414816 mode="download" [[ "${1:-}" == "--verify-only" ]] && mode="verify" diff --git a/scripts/frank-medium.env b/scripts/frank-medium.env index f0cd8d5..4c9dcc1 100644 --- a/scripts/frank-medium.env +++ b/scripts/frank-medium.env @@ -1,5 +1,5 @@ # Tunables for frank-medium (ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP -# UD-Q8_K_XL + magnitudedev/Qwen3.8-27B-DSpark-GGUF Q8_0 drafter), mainline llama.cpp +# UD-Q8_K_XL + z-lab/Qwen3.8-27B-DFlash2-GGUF Q8_0 drafter), mainline llama.cpp # on Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this # is the SOLE LLM runtime on the box: the vllm-radiance replica pair and its # routing proxy are retired. @@ -39,11 +39,11 @@ # carries the model's MTP head (blk.64.nextn.*), so a file-level switch to # --spec-type draft-mtp is available without a second copy in VRAM. # -# Drafter: magnitudedev/Qwen3.8-27B-DSpark-GGUF @ Q8_0, 1.27 GiB: 5 layers, -# hidden 5120, block-diffusion proposal with the Markov head (dspark). -# Decoding stays lossless -- the target verifies every drafted token. Its KV -# spans the whole context (see SIZING): unlike DFlash2 it declares no sliding -# window, and that is what the tensor split below pays for. +# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q8_0, 1.92 GiB: 5 layers, hidden +# 5120, a block-diffusion proposal (block_size 8) under a 2048-token sliding +# window. Decoding stays lossless -- the target verifies every drafted token. +# The window is what makes it cheap HERE: its KV is bounded at 2048 positions +# (~40 MiB) instead of spanning the context (see SIZING). # # Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet -> # FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose @@ -57,11 +57,21 @@ # of what is seen below -- they are allocated at load, not per request, and they # scale with --ubatch-size: # -# CTX split batch slots GPU 0 free GPU 1 free note -# 262144 1,1 2048 1 8.64 GiB 4.43 GiB DFlash2, day's start -# 262144 38,27 2048 1 5.49 GiB 3.41 GiB DSpark, 1 slot -# 262144 38,27 6144 2 3.78 GiB 0.08 GiB 3x batch is ASYMMETRIC -# 262144 40,25 6144 2 1.87 GiB 1.86 GiB <- deployed +# CTX split batch slots drafter GPU 0 free GPU 1 free note +# 262144 1,1 2048 1 DFlash2 8.64 GiB 4.43 GiB day's start +# 262144 39,26 6144 2 DFlash2 2.42 GiB 7.20 GiB +# 262144 38,27 6144 2 DFlash2 3.85 GiB 5.77 GiB +# 262144 37,28 6144 2 DFlash2 4.28 GiB 5.34 GiB +# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB <- deployed +# 262144 35,30 6144 2 DFlash2 5.14 GiB 4.48 GiB +# 262144 40,25 6144 2 DSpark 1.87 GiB 1.86 GiB (retired) +# +# ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE. This is a hybrid +# stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a +# DeltaNet layer shifts ~0.43 GiB while an attention layer shifts ~1.43 GiB +# (0.45 of weights plus ~1.0 GiB of its own f16 KV at 262144). Both step sizes +# were measured, which is why the balance is found by measurement rather than by +# interpolating between splits. # # 262144 is the window the model ships (n_ctx_train) and it fits with room, so # the fleet gets the whole thing; the arithmetic that used to cap this endpoint @@ -69,32 +79,30 @@ # adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers # rather than assuming, and prefer CTX_SIZE over --gpu-layers. # -# WHY THE SPLIT IS 40:25, AND WHY IT DEPENDS ON THE BATCH. Two effects pull in -# opposite directions: -# * THE DRAFT KEEPS ITS OWN FULL-CONTEXT KV ON ROCm1. The draft context is -# built with `n_ctx = llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so -# DSpark -- which, unlike DFlash2 (`sliding_window` 2048, ~40 MiB), declares -# no window -- caches 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per -# position: 5.0 GiB at 262144. At a 1:1 split that put ROCm1 at 33.9 GB of -# 34.2 GB and the draft's first graph capture died with `ROCm error: out of -# memory` (common_speculative_impl_draft_dflash::process). Hence 38:27. +# WHY THE SPLIT IS 36:29. Two effects both pull the balance BELOW an even +# split: +# * THE DRAFTER AND ITS OUTPUT LAYERS RIDE ROCm1. It carries the drafter's +# weights, the draft graph and the last target layers, so it must be given +# fewer. # * THE BIGGER BATCH COSTS ROCm1 ABOUT TWICE WHAT IT COSTS ROCm0 (~3.3 vs # ~1.7 GiB going 2048/512 -> 6144/1536 at 262144): ROCm1 holds the output -# layers AND the drafter's graphs. At 38:27 that left ROCm1 at 0.08 GiB -# free against ROCm0's 3.78 -- the reason for the rebalance. -# One -ts step shifts ~1.5 GiB of load between the cards at this batch, so the -# balance is found by measurement, not arithmetic. 40:25 is the measured -# balance (a 0.05 GiB gap) and it is also what MAXIMISES the context: with the -# floor at ~1.5 GiB per card, an unbalanced split would have to buy its -# headroom out of CTX_SIZE. +# layers AND the drafter's graphs. (Under the DSpark drafter that asymmetry +# dominated the picture, because DSpark also caches the WHOLE context; its +# KV alone was 5.0 GiB. DFlash2's window caps its KV at ~40 MiB, which +# freed ~4.3 GiB on ROCm1 and moved the balance four layers.) +# The layer-size granularity pushes back from the other side: 36:29 is the +# measured minimum (0.20 GiB gap, against 0.65 at 35:30 and 1.06 at 37:28). +# It also leaves the MOST ROOM -- this configuration clears the operator's +# 1.5 GiB-per-card floor by 3.2 GiB, so no headroom has to be bought out of +# CTX_SIZE and the full window stays. # # A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth # request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine # surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort, # i.e. a fault in the cross-card boundary copy, this layer split's hot path. It -# has not reproduced (38:27 served the whole e2e gate and a burst of requests; -# 40:25 has served this file's verification battery), but the signature is -# worth recognising -- it is NOT an OOM, and no amount of headroom prevents it. +# has not reproduced (36:29 has served this file's verification battery, and the +# e2e gate), but the signature is worth recognising -- it is NOT an OOM, and no +# amount of headroom prevents it. # # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads # those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp @@ -106,8 +114,8 @@ LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama # Payload, fetched with scripts/download-frank-medium-gguf.sh. MODEL_DIR=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF MODEL_FILE=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf -DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DSpark-GGUF -DRAFT_MODEL_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf +DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF +DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf # Fleet identity. 7770 is the fleet's dankserver endpoint; the alias is the id # at /v1/models and what every client asks for. @@ -121,11 +129,12 @@ HIP_VISIBLE_DEVICES=0,1 DEVICES=ROCm0,ROCm1 SPLIT_MODE=layer -# 40:25 -- the measured balance at the deployed batch, and the point that -# maximises CTX_SIZE: ROCm1 carries the drafter, its full-context KV and the -# output layers, so it takes fewer target layers. See SIZING above; one -ts -# step is worth ~1.5 GiB at this batch, so do not move it without re-measuring. -TENSOR_SPLIT=40,25 +# 36:29 -- the measured balance at the deployed batch, and the split that leaves +# the most room: ROCm1 carries the drafter, its KV and the output layers, so it +# takes fewer target layers. See SIZING above for the measurement table; one +# -ts step is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so +# do not move it without re-measuring. +TENSOR_SPLIT=36,29 NGL=999 # Auto-fit has no business resizing a context we set explicitly. @@ -142,8 +151,10 @@ PARALLEL=2 # 3x the build's defaults (2048/512). --ubatch-size is the physical batch the # graph is built at and therefore what the compute buffers cost: at 262144 this # setting needs ~1.7 GiB more on ROCm0 and ~3.3 GiB on ROCm1 than 2048/512 does -# -- which is what forced the rebalance above. 8192/2048 does not fit at all -# here: its 2020 MiB pp buffer fails to allocate on ROCm1. +# -- which is what put the balance below an even split. (8192/2048 failed to +# allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had +# 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and with +# DFlash2's ~4.9 GiB of headroom it has NOT been re-measured.) BATCH_SIZE=6144 UBATCH_SIZE=1536 @@ -181,13 +192,13 @@ REASONING_BUDGET=-1 CHAT_TEMPLATE_FILE= CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}' -# Speculative decoding: the DSpark block-diffusion drafter PLUS the draftless -# n-gram matcher. DSpark's GGUF carries block_size 7, so at most 6 draft +# Speculative decoding: the DFlash2 block-diffusion drafter PLUS the draftless +# n-gram matcher. DFlash2's GGUF carries block_size 8, so at most 7 draft # tokens per verification step (llama.cpp asserts n_max <= block_size - 1). # The drafter rides ROCm1, the card that ends up holding the target's last # layers. -SPEC_TYPE=draft-dspark,ngram-mod -SPEC_DRAFT_N_MAX=6 +SPEC_TYPE=draft-dflash,ngram-mod +SPEC_DRAFT_N_MAX=7 SPEC_DRAFT_NGL=999 SPEC_DRAFT_DEVICE=ROCm1 SPEC_DRAFT_TYPE_K=f16 diff --git a/scripts/serve-frank-medium.sh b/scripts/serve-frank-medium.sh index 5004a27..3afcaff 100755 --- a/scripts/serve-frank-medium.sh +++ b/scripts/serve-frank-medium.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Serve ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL as # frank-medium on Dankserver's two Radeon AI PRO R9700 (gfx1201), with the -# magnitudedev DSpark block-diffusion drafter, using mainline llama.cpp. +# z-lab DFlash2 block-diffusion drafter, using mainline llama.cpp. # # Runtime: llama.cpp master fb27a525 (2026-09-16), a gfx1201 HIP build that # bundles its own ROCm 7.14 runtime (RPATH $ORIGIN/../lib), produced by @@ -74,19 +74,17 @@ fi : "${CHAT_TEMPLATE_FILE:=}" : "${CHAT_TEMPLATE_KWARGS:={\"enable_thinking\":true,\"preserve_thinking\":true}}" -# Speculative decoding: draft-dflash (the DFlash2 drafter is a file, not a -# flag) or draft-mtp (the target GGUF's own MTP head -- no draft file). +# Speculative decoding: draft-dflash or draft-dspark (both are MODEL drafters -- +# a file, not a flag), or draft-mtp (the target GGUF's own MTP head, no draft +# file). The drafter's own tuning knobs (n-min, p-min, p-split, poll) are left +# at the build's defaults: they were only ever passed for dspark, and dspark is +# not the deployed drafter. : "${SPEC_TYPE:=draft-dflash}" : "${SPEC_DRAFT_N_MAX:=7}" -: "${SPEC_DRAFT_N_MIN:=0}" -: "${SPEC_DRAFT_P_MIN:=0.0}" -: "${SPEC_DRAFT_P_SPLIT:=0.10}" : "${SPEC_DRAFT_NGL:=999}" : "${SPEC_DRAFT_DEVICE:=}" : "${SPEC_DRAFT_TYPE_K:=f16}" : "${SPEC_DRAFT_TYPE_V:=f16}" -: "${SPEC_DRAFT_POLL:=1}" -: "${SPEC_DRAFT_POLL_BATCH:=1}" : "${SPEC_NGRAM_N_MATCH:=24}" : "${SPEC_NGRAM_N_MIN:=48}" : "${SPEC_NGRAM_N_MAX:=64}" @@ -129,9 +127,6 @@ if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then --spec-draft-type-v "$SPEC_DRAFT_TYPE_V" ) fi - if [[ "$SPEC_TYPE" == *dspark* || ( "$SPEC_DRAFT_P_MIN" != "0.0" && "$SPEC_DRAFT_P_MIN" != "0" ) ]]; then - SPEC_ARGS+=(--spec-draft-n-min "$SPEC_DRAFT_N_MIN" --spec-draft-p-min "$SPEC_DRAFT_P_MIN") - fi # ngram-mod is draftless: it needs no draft context and its window is its # own (--spec-ngram-mod-*), so it can ride in the chain beside a model # drafter. It also costs no VRAM -- a CPU hash table. diff --git a/tests/frank_medium_unit.bats b/tests/frank_medium_unit.bats index 4efec3f..79c3677 100755 --- a/tests/frank_medium_unit.bats +++ b/tests/frank_medium_unit.bats @@ -5,7 +5,7 @@ # day. What this seam pins is what the flags alone do not say: boot persistence # for a NATIVE runtime (no generated quadlet is involved), that BOTH cards work # on ONE model through a layer split with no collective in the path, that the -# DSpark drafter is wired with a depth its block size actually allows, that the +# DFlash2 drafter is wired with a depth its block size actually allows, that the # runtime and payload are pinned rather than floated, and that one slot's KV # still fits a 32 GiB card beside the weights it holds. @@ -21,7 +21,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" # Measured constants (rocm-smi / HF blob sizes), used by the fit arithmetic. CARD_BYTES=34208743424 # one R9700 as rocm-smi reports it MODEL_BYTES=31457990112 # UD-Q8_K_XL target -DRAFT_BYTES=1359284737 # DSpark Q8_0 drafter +DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter @test "frank-medium's unit, serve script, env and its build/fetch scripts exist" { [ -f "$UNIT" ] @@ -47,7 +47,7 @@ DRAFT_BYTES=1359284737 # DSpark Q8_0 drafter @test "both R9700s serve ONE model, layer-split inside one process" { grep -q '^SPLIT_MODE=layer$' "$ENV_FILE" grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE" - grep -q '^TENSOR_SPLIT=40,25$' "$ENV_FILE" + grep -q '^TENSOR_SPLIT=36,29$' "$ENV_FILE" grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE" grep -q -- '--device "$DEVICES"' "$SERVE" grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE" @@ -65,8 +65,8 @@ DRAFT_BYTES=1359284737 # DSpark Q8_0 drafter # The chain is the measured winner (see the runbook's A/B): free where the # matcher cannot match, ~2.6x the drafter alone on prompt-echo traffic. A # bare `draft-dflash` here would silently give that up. - grep -qE '^SPEC_TYPE=draft-dspark,ngram-mod$' "$ENV_FILE" - grep -q '^DRAFT_MODEL_FILE=Qwen3\.8-27B-DSpark-Q8_0\.gguf$' "$ENV_FILE" + grep -qE '^SPEC_TYPE=draft-dflash,ngram-mod$' "$ENV_FILE" + grep -q '^DRAFT_MODEL_FILE=Qwen3\.8-27B-DFlash2-Q8_0\.gguf$' "$ENV_FILE" grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE" grep -q -- '--spec-draft-model "$DRAFT_MODEL"' "$SERVE" # ngram-mod's own window -- and only when the type asks for it, so a pure @@ -85,11 +85,11 @@ end = next(i for i in range(guard + 1, len(lines)) if lines[i].strip() == 'fi' and len(lines[i]) - len(lines[i].lstrip()) == indent) assert '--spec-ngram-mod-n-match' in chr(10).join(lines[guard:end]), 'ngram flags must sit in the ngram branch' " - # DSpark drafts into a block of 7, so at most 6 draft tokens per + # DFlash2 drafts into a block of 8, so at most 7 draft tokens per # verification step -- llama.cpp asserts n_max <= block_size - 1. local n_max n_max="$(sed -nE 's/^SPEC_DRAFT_N_MAX=([0-9]+)$/\1/p' "$ENV_FILE")" - [ -n "$n_max" ] && [ "$n_max" -ge 1 ] && [ "$n_max" -le 6 ] + [ -n "$n_max" ] && [ "$n_max" -ge 1 ] && [ "$n_max" -le 7 ] } @test "a draft MODEL is passed only for DFlash types, never for MTP" { @@ -124,7 +124,7 @@ print('draft-model wiring ok') @test "the payload is fetched resumably and verified by size" { grep -q 'UD-Q8_K_XL\.gguf' "$FETCH" - grep -q 'Qwen3\.8-27B-DSpark-Q8_0\.gguf' "$FETCH" + grep -q 'Qwen3\.8-27B-DFlash2-Q8_0\.gguf' "$FETCH" grep -q 'curl -fL -C -' "$FETCH" grep -q 'verifying sizes' "$FETCH" } @@ -144,25 +144,32 @@ print('draft-model wiring ok') [ "$parallel" -ge 1 ] [ "$batch" -ge "$ubatch" ] [ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ] - # Per card, calibrated against rocm-smi at the deployed point (40:25, ctx - # 262144, batch 6144 x ubatch 1536, two slots, DSpark) where the two cards - # measured 32.199e9 / 32.212e9 B used: + # Per card, calibrated against rocm-smi at the deployed point (36:29, ctx + # 262144, batch 6144 x ubatch 1536, two slots, DFlash2), where the two cards + # measured 4.71 / 4.91 GiB free. The fit lands within ~0.05 GiB of that. # # card 0 = its share of the weights and of the KV (qwen35 spends KV on # its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim # x 2 B = 64 KiB per token at f16) plus ~1.4e6 B per ubatch unit # of compute buffers - # card 1 = the same for its own share, plus the WHOLE drafter -- its - # weights AND its full-context KV, because DSpark declares no - # sliding window (5 x 2 x 8 x 128 x 2 B = 20480 B per position; - # the draft context is sized from the target's: - # `n_ctx = llama_n_ctx(ctx_tgt)`) -- and ~2.7e6 B per ubatch unit - # on top of a ~2.3 GiB constant. Its buffers are the heavier ones - # because it holds the output layers and the drafter's graphs. + # card 1 = the same for its own share, plus the WHOLE drafter: its weights + # (2.06e9 B), its KV -- BOUNDED BY ITS SLIDING WINDOW, so + # 20480 B/pos x 2048 positions rather than the context -- and + # ~2.7e6 B per ubatch unit of buffers on top of a ~4.8 GiB + # constant. Its buffers are the heavier ones because it holds the + # output layers and the drafter's graphs. # - # Both cards must clear the operator's floor, and a change to ctx, ubatch or - # the split has to keep them from drifting apart -- an unbalanced split is - # what cost ROCm1 3.7 GiB of headroom when the batch tripled (2026-09-16). + # The layer it charges for is the AVERAGE (0.70 GiB: 31.46e9 of weights plus + # 16 GiB of KV over 65 layers). This stack is a hybrid, so real steps vary + # around that -- 0.43 GiB for a DeltaNet layer, 1.43 GiB for a full-attention + # one -- which makes the drift check below CONSERVATIVE: it overstates the + # already trips it. That is the intent: moving the split means re-measuring, + # not re-arguing. + # + # The ubatch coefficients ARE fitted at 1536, so the floor check's "an + # 8192/2048 batch does not fit" is an EXTRAPOLATION, not a measurement -- + # and it cannot see CONTIGUITY at all, which is the actual mechanism by which + # the DSpark-era 8192/2048 failed (a 2020 MiB pp buffer that found no room). local kv_pos=65536 draft_kv_pos=20480 floor slack local card_hi card_lo diff floor=$(( 1536 * 1024 * 1024 )) # 1.5 GiB per card @@ -173,8 +180,8 @@ print('draft-model wiring ok') card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \ + DRAFT_BYTES \ + kv_pos * ctx * split_lo / (split_hi + split_lo) \ - + draft_kv_pos * ctx \ - + 2353 * 1024 * 1024 \ + + draft_kv_pos * (ctx < 2048 ? ctx : 2048) \ + + 828697861 \ + 2740 * 1024 * ubatch )) [ "$card_hi" -le $(( CARD_BYTES - floor )) ] [ "$card_lo" -le $(( CARD_BYTES - floor )) ]