Frank
91546dbb95
DSpark came off after a day. It was the better-looking file (1.27 GiB against 1.92) but it declares no sliding window, and the draft context is built with n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window (5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the draft-carrying card. The balance point moved with it, so it was re-measured at the deployed batch (6144 x 1536, 2 slots, 262144) rather than interpolated: 39,26 2.42 / 7.20 37,28 4.28 / 5.34 35,30 5.14 / 4.48 38,27 3.85 / 5.77 36,29 4.71 / 4.91 34,31 6.57 / 3.05 36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's 1.5 GiB floor on each card, so the full window stays. One -ts step is one layer, and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary, 1.43 GiB on a full-attention one -- which is why the table exists. Also in here: - the fetch script pointed at the wrong file (DSpark) AND declared the wrong size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made --verify-only fail on a payload that was in fact correct. Repointed at z-lab DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the Hub's LFS oids (host == Hub for both). - the seam's fit was recalibrated for a window-bounded draft KV and for the new point; it now passes 36:29 and fails every other split measured, including 1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which the comment says out loud. - the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll) are gone; the build's defaults apply.
264 lines
13 KiB
Bash
264 lines
13 KiB
Bash
# Tunables for frank-medium (ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP
|
|
# UD-Q8_K_XL + z-lab/Qwen3.8-27B-DFlash2-GGUF Q8_0 drafter), mainline llama.cpp
|
|
# on Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this
|
|
# is the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
|
|
# routing proxy are retired.
|
|
#
|
|
# Hardware: two R9700 (Navi 48, RDNA4, 32,624 MiB each), no XGMI, different
|
|
# IOMMU groups -- every cross-card byte goes over PCIe. The Raphael iGPU
|
|
# (gfx1036/Radeon 610M) enumerates as HIP device 2 and is hidden below so
|
|
# nothing can drift onto shared system RAM.
|
|
#
|
|
# WHY LAYER SPLIT AND NOT TENSOR (operator call, 2026-09-16): tensor parallel
|
|
# needs a cross-card allreduce per layer, and this host cannot dispatch one.
|
|
# RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr refuses it
|
|
# on a device without PCIe atomics -- the second card sits behind the B650
|
|
# chipset bridges (AtomicOpsCap: Routing-, read-only), so every collective that
|
|
# touches it dies with hipErrorIllegalState. Measured on this box 2026-09-15
|
|
# (llama-bench, pp4096 / tg128, f16 KV, fa on):
|
|
#
|
|
# single R9700 1160.9 / 19.42
|
|
# layer 2x 1991.2 / 18.76 (859.1 / 17.64 at depth 32768)
|
|
# tensor 2x 271.8 / 26.26 (no RCCL: butterfly fallback, -7x prefill)
|
|
#
|
|
# Layer split distributes layers in order; only the activation at the boundary
|
|
# crosses the wire, once per forward pass. That is the "pipeline parallelism"
|
|
# this service wanted, and it needs no collective at all.
|
|
#
|
|
# Runtime: mainline llama.cpp master fb27a525 (2026-09-16), gfx1201, built by
|
|
# scripts/build-llama-rdna4.sh against a private ROCm 7.14.0a20260624 toolchain
|
|
# that is bundled next to the binary (RPATH $ORIGIN/../lib, no LD_LIBRARY_PATH).
|
|
# The ref is pinned by TWO features: DFlash2 (ggml-org/llama.cpp#27342 -- the
|
|
# drafter is detected from its GGUF) and qwen35 MTP. Do not float it.
|
|
#
|
|
# Model: ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL,
|
|
# 29.29 GiB, arch qwen35, Unsloth Dynamic 3.0 (per-tensor types from Unsloth's
|
|
# own quant of the same size, on Unsloth's importance matrix). Abliteration is
|
|
# orcarouter's single refusal direction projected out of 131 tensors; the vision
|
|
# tower is untouched but unused here (text only, see NO_MMPROJ). The file also
|
|
# carries the model's MTP head (blk.64.nextn.*), so a file-level switch to
|
|
# --spec-type draft-mtp is available without a second copy in VRAM.
|
|
#
|
|
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q8_0, 1.92 GiB: 5 layers, hidden
|
|
# 5120, a block-diffusion proposal (block_size 8) under a 2048-token sliding
|
|
# window. Decoding stays lossless -- the target verifies every drafted token.
|
|
# The window is what makes it cheap HERE: its KV is bounded at 2048 positions
|
|
# (~40 MiB) instead of spanning the context (see SIZING).
|
|
#
|
|
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
|
|
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
|
|
# per-sequence state is CONSTANT-SIZE, 16 full-attention layers at 4 KV heads x
|
|
# 256 head_dim. A token therefore costs 2 x 4 x 256 x 2 B = 64 KiB of KV at f16
|
|
# in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet
|
|
# state at any depth.
|
|
#
|
|
# SIZING, measured (rocm-smi, 2026-09-16). Weights split across the two cards;
|
|
# the drafter AND its KV ride the second. The loader's compute buffers are part
|
|
# of what is seen below -- they are allocated at load, not per request, and they
|
|
# scale with --ubatch-size:
|
|
#
|
|
# CTX split batch slots drafter GPU 0 free GPU 1 free note
|
|
# 262144 1,1 2048 1 DFlash2 8.64 GiB 4.43 GiB day's start
|
|
# 262144 39,26 6144 2 DFlash2 2.42 GiB 7.20 GiB
|
|
# 262144 38,27 6144 2 DFlash2 3.85 GiB 5.77 GiB
|
|
# 262144 37,28 6144 2 DFlash2 4.28 GiB 5.34 GiB
|
|
# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB <- deployed
|
|
# 262144 35,30 6144 2 DFlash2 5.14 GiB 4.48 GiB
|
|
# 262144 40,25 6144 2 DSpark 1.87 GiB 1.86 GiB (retired)
|
|
#
|
|
# ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE. This is a hybrid
|
|
# stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a
|
|
# DeltaNet layer shifts ~0.43 GiB while an attention layer shifts ~1.43 GiB
|
|
# (0.45 of weights plus ~1.0 GiB of its own f16 KV at 262144). Both step sizes
|
|
# were measured, which is why the balance is found by measurement rather than by
|
|
# interpolating between splits.
|
|
#
|
|
# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
|
|
# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
|
|
# at 114688 was the vLLM predecessor's, not this runtime's. If a future change
|
|
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
|
|
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
|
|
#
|
|
# WHY THE SPLIT IS 36:29. Two effects both pull the balance BELOW an even
|
|
# split:
|
|
# * THE DRAFTER AND ITS OUTPUT LAYERS RIDE ROCm1. It carries the drafter's
|
|
# weights, the draft graph and the last target layers, so it must be given
|
|
# fewer.
|
|
# * THE BIGGER BATCH COSTS ROCm1 ABOUT TWICE WHAT IT COSTS ROCm0 (~3.3 vs
|
|
# ~1.7 GiB going 2048/512 -> 6144/1536 at 262144): ROCm1 holds the output
|
|
# layers AND the drafter's graphs. (Under the DSpark drafter that asymmetry
|
|
# dominated the picture, because DSpark also caches the WHOLE context; its
|
|
# KV alone was 5.0 GiB. DFlash2's window caps its KV at ~40 MiB, which
|
|
# freed ~4.3 GiB on ROCm1 and moved the balance four layers.)
|
|
# The layer-size granularity pushes back from the other side: 36:29 is the
|
|
# measured minimum (0.20 GiB gap, against 0.65 at 35:30 and 1.06 at 37:28).
|
|
# It also leaves the MOST ROOM -- this configuration clears the operator's
|
|
# 1.5 GiB-per-card floor by 3.2 GiB, so no headroom has to be bought out of
|
|
# CTX_SIZE and the full window stays.
|
|
#
|
|
# A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth
|
|
# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
|
|
# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
|
|
# i.e. a fault in the cross-card boundary copy, this layer split's hot path. It
|
|
# has not reproduced (36:29 has served this file's verification battery, and the
|
|
# e2e gate), but the signature is worth recognising -- it is NOT an OOM, and no
|
|
# amount of headroom prevents it.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
|
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
|
# as flags only.
|
|
|
|
# llama-server: mainline fb27a525 built for gfx1201 by scripts/build-llama-rdna4.sh.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server
|
|
|
|
# Payload, fetched with scripts/download-frank-medium-gguf.sh.
|
|
MODEL_DIR=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
|
MODEL_FILE=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
|
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF
|
|
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf
|
|
|
|
# Fleet identity. 7770 is the fleet's dankserver endpoint; the alias is the id
|
|
# at /v1/models and what every client asks for.
|
|
ALIAS=frank-medium
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7770
|
|
# Device placement. HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2)
|
|
# so nothing can drift onto shared system RAM; DEVICES is what llama.cpp splits
|
|
# across, and neither of the two cards has an XGMI link to the other.
|
|
HIP_VISIBLE_DEVICES=0,1
|
|
DEVICES=ROCm0,ROCm1
|
|
SPLIT_MODE=layer
|
|
|
|
# 36:29 -- the measured balance at the deployed batch, and the split that leaves
|
|
# the most room: ROCm1 carries the drafter, its KV and the output layers, so it
|
|
# takes fewer target layers. See SIZING above for the measurement table; one
|
|
# -ts step is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so
|
|
# do not move it without re-measuring.
|
|
TENSOR_SPLIT=36,29
|
|
|
|
NGL=999
|
|
# Auto-fit has no business resizing a context we set explicitly.
|
|
FIT=off
|
|
FLASH_ATTN=on
|
|
|
|
# The model's full window (n_ctx_train 262144) divided over TWO slots, so each
|
|
# conversation gets 131072 -- verified at load as `n_slots = 2, n_ctx_slot =
|
|
# 131072`. --parallel DIVIDES --ctx-size, so more slots would shrink every
|
|
# conversation rather than grow the cache.
|
|
CTX_SIZE=262144
|
|
PARALLEL=2
|
|
|
|
# 3x the build's defaults (2048/512). --ubatch-size is the physical batch the
|
|
# graph is built at and therefore what the compute buffers cost: at 262144 this
|
|
# setting needs ~1.7 GiB more on ROCm0 and ~3.3 GiB on ROCm1 than 2048/512 does
|
|
# -- which is what put the balance below an even split. (8192/2048 failed to
|
|
# allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had
|
|
# 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and with
|
|
# DFlash2's ~4.9 GiB of headroom it has NOT been re-measured.)
|
|
BATCH_SIZE=6144
|
|
UBATCH_SIZE=1536
|
|
|
|
# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
|
|
# f16 K/V: an agentic checkpoint is the wrong place to spend KV precision.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
# Context checkpoints: 0 is the build's default -- no per-slot KV snapshots, so
|
|
# a re-sent or rewound conversation re-prefills instead of rewinding. Stated
|
|
# explicitly so the tunable stays visible rather than merely absent.
|
|
CTX_CHECKPOINTS=0
|
|
CACHE_RAM=8192
|
|
|
|
# Sampling defaults for requests that send none -- the values the vLLM
|
|
# predecessor served on this endpoint (`--override-generation-config`), so a
|
|
# client sees no change across the runtime swap. The model card's own
|
|
# recommendation is temp 1.0 / top_p 0.95; the harness sets its own per model in
|
|
# models.yml, so this is the floor, not the contract.
|
|
TEMPERATURE=0.7
|
|
TOP_P=0.95
|
|
TOP_K=20
|
|
MIN_P=0.0
|
|
PRESENCE_PENALTY=0.0
|
|
|
|
# Reasoning on, projected into reasoning_content (deepseek format) with no
|
|
# budget: the model decides how long to think.
|
|
REASONING=on
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|
|
|
|
# Chat template: empty = the one embedded in the GGUF (Qwen3.8's own).
|
|
CHAT_TEMPLATE_FILE=
|
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
|
|
|
# Speculative decoding: the DFlash2 block-diffusion drafter PLUS the draftless
|
|
# n-gram matcher. DFlash2's GGUF carries block_size 8, so at most 7 draft
|
|
# tokens per verification step (llama.cpp asserts n_max <= block_size - 1).
|
|
# The drafter rides ROCm1, the card that ends up holding the target's last
|
|
# layers.
|
|
SPEC_TYPE=draft-dflash,ngram-mod
|
|
SPEC_DRAFT_N_MAX=7
|
|
SPEC_DRAFT_NGL=999
|
|
SPEC_DRAFT_DEVICE=ROCm1
|
|
SPEC_DRAFT_TYPE_K=f16
|
|
SPEC_DRAFT_TYPE_V=f16
|
|
|
|
# MTP (the model's own head) vs the drafter -- MEASURED 2026-09-16 (temperature
|
|
# 0, 1k prompt / 512 out; the autoregressive baseline and both MTP depths were
|
|
# run against the deployed configuration in one session, same binary, same
|
|
# prompts):
|
|
#
|
|
# none 17.7 tok/s card1 26.07 GB
|
|
# draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
|
|
# draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
|
|
# draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
|
|
#
|
|
# All four produced IDENTICAL greedy text, so verification is lossless whichever
|
|
# proposer runs. DFlash2 is worth 2.46x over autoregressive and 34 % over the
|
|
# best MTP; deeper MTP drafting costs more than it returns (n=7 accepts 33 % of
|
|
# 147 proposals, n=3 accepts 58 % of 72). The MTP head is not free either:
|
|
# selecting it loads blk.64.nextn.* and adds ~3.3 GB to the card that holds it,
|
|
# against 2.06 GB for the drafter's file. Do not switch this without
|
|
# re-measuring.
|
|
#
|
|
# THE TWO CANNOT BE CHAINED. `--spec-type draft-mtp,draft-dflash` dies at load:
|
|
# context type MTP requested but model doesn't contain MTP layers
|
|
# failed to create MTP context
|
|
# A chain has ONE draft context, and MTP builds against whatever model that
|
|
# context holds -- here the DFlash2 file, which has no MTP layers. MTP's head is
|
|
# only loaded when MTP IS the draft model ("creating MTP draft context against
|
|
# the target model ..."), so a chain of the two is a structural dead end, not a
|
|
# tuning question. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md.
|
|
|
|
# ngram-mod: the draftless n-gram matcher (CPU hash table in host RAM, no VRAM,
|
|
# no model file, no draft context). It joins the chain unconditionally --
|
|
# unlike MTP, which needs a draft context and therefore cannot share one with
|
|
# DFlash2. Verified on the loaded server: with ngram-mod as the ONLY spec type,
|
|
# a prose prompt produces ZERO drafts (plain autoregressive) while a
|
|
# prompt-echo prompt accepts 100 % of them, so its drafts are unmistakable.
|
|
#
|
|
# WINDOW MEASURED 2026-09-16 (chain at each window, decode tok/s):
|
|
#
|
|
# n_min/n_max prose 1k/512 prompt-echo 1.5k/400
|
|
# 48/64 43.9 200.6 <- deployed
|
|
# 16/32 36.0 161.0
|
|
# 8/16 36.2 113.8
|
|
#
|
|
# The long window is the right one, and not only because it wins on echo: a
|
|
# SHORT window makes the matcher propose on short fragments of ordinary prose,
|
|
# where the proposals are rejected and the verification still costs -- which is
|
|
# the "ngram hurts" result the fleet measured on other payloads twice before.
|
|
# With 48/64 the chain is byte-identical to DFlash2 alone on prose (same 119
|
|
# drafts / 49 accepted) and on the identity prompt (70/53).
|
|
#
|
|
# Do not shorten these without re-running scripts/bench-spec-ab.py: the
|
|
# no-penalty property is what makes the chain safe as a default.
|
|
SPEC_NGRAM_N_MATCH=24
|
|
SPEC_NGRAM_N_MIN=48
|
|
SPEC_NGRAM_N_MAX=64
|
|
|
|
# Text only: the server must not autoload a vision tower. Vision stays on
|
|
# frank-small (Dankclaw, RTX 3090), which is the fleet's image endpoint.
|
|
NO_MMPROJ=1
|