Frank
23f09788e9
DFlash2's sidecar (1.92 GiB + buffers on ROCm1) kept the 393216 pool off the table; the MTP head lives in the target file, so its cost is one nextn layer's KV (4096 B/pos) plus small buffers. Fit: ROCm0 29.77 / ROCm1 28.32 GiB, over the floor, 34:31 the balance point at this pool. n_max 3 is the measured MTP winner. The seam's fit is now drafter-aware, the slack is the deployed integer gap (1483 MiB), the ctx cap 393216 with the rope caveat (top 131k extrapolates on the base rope).
312 lines
16 KiB
Bash
312 lines
16 KiB
Bash
# Tunables for frank-medium (ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP
|
|
# UD-Q8_K_XL + z-lab/Qwen3.8-27B-DFlash2-GGUF Q8_0 drafter), mainline llama.cpp
|
|
# on Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this
|
|
# is the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
|
|
# routing proxy are retired.
|
|
#
|
|
# Hardware: two R9700 (Navi 48, RDNA4, 32,624 MiB each), no XGMI, different
|
|
# IOMMU groups -- every cross-card byte goes over PCIe. The Raphael iGPU
|
|
# (gfx1036/Radeon 610M) enumerates as HIP device 2 and is hidden below so
|
|
# nothing can drift onto shared system RAM.
|
|
#
|
|
# WHY LAYER SPLIT AND NOT TENSOR (operator call, 2026-09-16): tensor parallel
|
|
# needs a cross-card allreduce per layer, and this host cannot dispatch one.
|
|
# RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr refuses it
|
|
# on a device without PCIe atomics -- the second card sits behind the B650
|
|
# chipset bridges (AtomicOpsCap: Routing-, read-only), so every collective that
|
|
# touches it dies with hipErrorIllegalState. Measured on this box 2026-09-15
|
|
# (llama-bench, pp4096 / tg128, f16 KV, fa on):
|
|
#
|
|
# single R9700 1160.9 / 19.42
|
|
# layer 2x 1991.2 / 18.76 (859.1 / 17.64 at depth 32768)
|
|
# tensor 2x 271.8 / 26.26 (no RCCL: butterfly fallback, -7x prefill)
|
|
#
|
|
# Layer split distributes layers in order; only the activation at the boundary
|
|
# crosses the wire, once per forward pass. That is the "pipeline parallelism"
|
|
# this service wanted, and it needs no collective at all.
|
|
#
|
|
# Runtime: mainline llama.cpp master fb27a525 (2026-09-16), gfx1201, built by
|
|
# scripts/build-llama-rdna4.sh against a private ROCm 7.14.0a20260624 toolchain
|
|
# that is bundled next to the binary (RPATH $ORIGIN/../lib, no LD_LIBRARY_PATH).
|
|
# The ref is pinned by TWO features: DFlash2 (ggml-org/llama.cpp#27342 -- the
|
|
# drafter is detected from its GGUF) and qwen35 MTP. Do not float it.
|
|
#
|
|
# Model: ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL,
|
|
# 29.29 GiB, arch qwen35, Unsloth Dynamic 3.0 (per-tensor types from Unsloth's
|
|
# own quant of the same size, on Unsloth's importance matrix). Abliteration is
|
|
# orcarouter's single refusal direction projected out of 131 tensors; the vision
|
|
# tower loads from mmproj-BF16.gguf (see NO_MMPROJ). The file also
|
|
# carries the model's MTP head (blk.64.nextn.*), so a file-level switch to
|
|
# --spec-type draft-mtp is available without a second copy in VRAM.
|
|
#
|
|
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q8_0, 1.92 GiB: 5 layers, hidden
|
|
# 5120, a block-diffusion proposal (block_size 8) under a 2048-token sliding
|
|
# window. Decoding stays lossless -- the target verifies every drafted token.
|
|
# The window is what makes it cheap HERE: its KV is bounded at 2048 positions
|
|
# (~40 MiB) instead of spanning the context (see SIZING).
|
|
#
|
|
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
|
|
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
|
|
# per-sequence state is CONSTANT-SIZE, 16 full-attention layers at 4 KV heads x
|
|
# 256 head_dim. A token therefore costs 2 x 4 x 256 x 2 B = 64 KiB of KV at f16
|
|
# in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet
|
|
# state at any depth.
|
|
#
|
|
# SIZING, measured (rocm-smi, 2026-09-16). Weights split across the two cards;
|
|
# the drafter AND its KV ride the second. The loader's compute buffers are part
|
|
# of what is seen below -- they are allocated at load, not per request, and they
|
|
# scale with --ubatch-size:
|
|
#
|
|
# CTX split batch slots drafter GPU 0 free GPU 1 free note (ubatch 1536)
|
|
# 262144 1,1 2048 1 DFlash2 8.64 GiB 4.43 GiB day's start
|
|
# 262144 39,26 6144 2 DFlash2 2.42 GiB 7.20 GiB
|
|
# 262144 38,27 6144 2 DFlash2 3.85 GiB 5.77 GiB
|
|
# 262144 37,28 6144 2 DFlash2 4.28 GiB 5.34 GiB
|
|
# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB
|
|
# 262144 35,30 6144 2 DFlash2 5.14 GiB 4.48 GiB
|
|
# 262144 40,25 6144 2 DSpark 1.87 GiB 1.86 GiB (retired)
|
|
#
|
|
# ...and the same deployed shape re-measured at ubatch 512, after llama-bench
|
|
# showed 512 is the throughput optimum and costs 3.71 GiB less in total (see the
|
|
# batch block below):
|
|
#
|
|
# 262144 36,29 6144 512 2 DFlash2 6.01 GiB 7.32 GiB ubatch only
|
|
# 262144 35,30 6144 512 2 DFlash2 6.44 GiB 6.90 GiB <- deployed
|
|
#
|
|
# ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE. This is a hybrid
|
|
# stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a
|
|
# DeltaNet layer shifts ~0.43 GiB while an attention layer shifts ~1.43 GiB
|
|
# (0.45 of weights plus ~1.0 GiB of its own f16 KV at 262144). Both step sizes
|
|
# were measured, which is why the balance is found by measurement rather than by
|
|
# interpolating between splits.
|
|
#
|
|
# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
|
|
# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
|
|
# at 114688 was the vLLM predecessor's, not this runtime's. If a future change
|
|
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
|
|
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
|
|
#
|
|
# WHY THE SPLIT IS 35:30. Three effects decide it, and all three are measured:
|
|
# * THE DRAFTER AND ITS OUTPUT LAYERS RIDE ROCm1. It carries the drafter's
|
|
# weights, the draft graph and the last target layers, so it takes fewer
|
|
# than half the layer count.
|
|
# * THE UBATCH BUFFERS COST ROCm1 MORE THAN ROCm0, and shrinking them pays
|
|
# ROCm1 back more: 1536 -> 512 freed 1.30 GiB on ROCm0 but 2.41 GiB on
|
|
# ROCm1 at this shape. Read the other way, that is exactly why the smaller
|
|
# batch moved the balance toward ROCm1 -- and cost ROCm0 one layer.
|
|
# * THE LAYER GRANULARITY IS THE FLOOR ON HOW WELL ANY SPLIT CAN BALANCE. One
|
|
# -ts step is one layer -- ~0.43 GiB DeltaNet, ~1.43 GiB attention -- so the
|
|
# reachable minimum gap is comparable to one step rather than to zero. 35:30
|
|
# sits at 0.46 GiB, i.e. one DeltaNet layer's worth, against 1.31 GiB at
|
|
# 36:29.
|
|
# It also leaves the most room: 6.44 / 6.90 GiB free clears the operator's 1.5
|
|
# GiB-per-card floor by ~5 GiB, so no headroom has to be bought out of CTX_SIZE
|
|
# and the full window stays.
|
|
#
|
|
# A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth
|
|
# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
|
|
# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
|
|
# i.e. a fault in the cross-card boundary copy, this layer split's hot path. It
|
|
# has not reproduced (36:29 served this file's verification battery and the e2e
|
|
# gate, and 35:30 is the deployed shape), but the signature is worth recognising
|
|
# -- it is NOT an OOM, and no amount of headroom prevents it.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
|
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
|
# as flags only.
|
|
|
|
# llama-server: mainline fb27a525 built for gfx1201 by scripts/build-llama-rdna4.sh.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server
|
|
|
|
# Payload, fetched with scripts/download-frank-medium-gguf.sh.
|
|
MODEL_DIR=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
|
MODEL_FILE=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
|
# The BF16 vision projector, same repo as the target. llama-server autoloads
|
|
# any mmproj*.gguf beside the model; the serve script refuses to start without
|
|
# it when vision is enabled, so a missing file is a deploy error, not a
|
|
# text-only server that pretends to be multimodal.
|
|
MM_PROJ_FILE=mmproj-BF16.gguf
|
|
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF
|
|
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf
|
|
|
|
# Fleet identity. 7770 is the fleet's dankserver endpoint; the alias is the id
|
|
# at /v1/models and what every client asks for.
|
|
ALIAS=frank-medium
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7770
|
|
# Device placement. HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2)
|
|
# so nothing can drift onto shared system RAM; DEVICES is what llama.cpp splits
|
|
# across, and neither of the two cards has an XGMI link to the other.
|
|
HIP_VISIBLE_DEVICES=0,1
|
|
DEVICES=ROCm0,ROCm1
|
|
SPLIT_MODE=layer
|
|
|
|
# 34:31 -- the balance point at the 393216 pool (fit gap 1.44 GiB; MTP's head
|
|
# and its KV ride ROCm1 with the last layers, so the split sends MORE layers
|
|
# to ROCm0 than the 36:29 of the DFlash2/327680 configuration). One -ts step
|
|
# is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so do not
|
|
# move it without re-measuring.
|
|
TENSOR_SPLIT=34,31
|
|
|
|
NGL=999
|
|
# Auto-fit has no business resizing a context we set explicitly.
|
|
FIT=off
|
|
FLASH_ATTN=on
|
|
|
|
# 393216 positions over TWO slots, so each conversation gets 196608 --
|
|
# --parallel DIVIDES --ctx-size, so more slots would shrink every
|
|
# conversation rather than grow the cache. The checkpoint's trained window is
|
|
# 262144 and the GGUF carries NO rope scaling (freq_base 1e7, no YaRN keys),
|
|
# so positions beyond 262144 extrapolate on the base rope: it runs, and 1.5x
|
|
# is a modest ask of a rope, but the top 131k of the pool is untrained
|
|
# territory.
|
|
CTX_SIZE=393216
|
|
PARALLEL=2
|
|
|
|
# MEASURED 2026-09-16 with scripts/bench-batch-ubatch.sh on this box: llama-bench,
|
|
# a 64k prefill plus a 128-token decode with 64k of KV already resident, 3 reps,
|
|
# one invocation per cell, the deployed split and f16 KV. --ubatch-size is the
|
|
# physical batch the graph is built at -- and the only one of the two knobs that
|
|
# costs VRAM:
|
|
#
|
|
# batch ubatch prefill tok/s GPU 0 / GPU 1 free
|
|
# 6144 1536 1164.96 +- 2.30 10.28 / 11.71 GiB
|
|
# 6144 768 1135.03 +- 1.09 11.29 / 13.28 GiB
|
|
# 6144 512 1192.93 +- 0.91 11.63 / 13.81 GiB <- deployed
|
|
# 6144 384 1068.04 +- 0.79 11.80 / 14.07 GiB
|
|
# 6144 256 1045.98 +- 1.69 11.97 / 14.33 GiB
|
|
# 4096 1536 1161.90 +- 0.96 10.28 / 11.71 GiB
|
|
# 2048 1536 1170.23 +- 0.86 10.28 / 11.71 GiB
|
|
# 2048 512 1193.88 +- 1.01 11.63 / 13.81 GiB
|
|
#
|
|
# THE PLATEAU STARTS AT 512. 512 -> 1536 buys ~2 % of prefill -- inside the
|
|
# run-to-run spread, and the 512 cells were the faster pair in BOTH batch
|
|
# columns -- while costing 1.35 GiB on ROCm0 and 2.10 GiB on ROCm1, so the
|
|
# deployment takes the cheap end. BELOW 512 the SMs are no longer saturated:
|
|
# 384 and 256 fall ~11 % and ~12 %. The logical batch is free either way --
|
|
# 2048, 4096 and 6144 land within 2 % of each other at a fixed ubatch and use
|
|
# the SAME VRAM, because the compute buffers are sized by ubatch alone.
|
|
#
|
|
# The prefill figure is a property of the SHAPE, not of this knob: the same
|
|
# sweep with no depth in front of it (depth 0) runs 1543 tok/s at ubatch 256 and
|
|
# 1723 at 1536 -- the plateau is in the same place, and the ~32 % between the two
|
|
# tables is what attending over 64k of resident KV costs. Decode is 15.88 tok/s
|
|
# at depth 65536 on EVERY ubatch (17.80 at depth 0), so this knob cannot touch
|
|
# generation at all.
|
|
#
|
|
# 8192/2048 stays untested under this drafter (the "2020 MiB pp buffer fails on
|
|
# ROCm1" note was measured under DSpark, when it had 0.08 GiB free), and there is
|
|
# no longer a reason to want it: batch buys nothing measurable here.
|
|
BATCH_SIZE=6144
|
|
UBATCH_SIZE=512
|
|
|
|
# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
|
|
# f16 K/V: an agentic checkpoint is the wrong place to spend KV precision.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
# Context checkpoints: 0 is the build's default -- no per-slot KV snapshots, so
|
|
# a re-sent or rewound conversation re-prefills instead of rewinding. Stated
|
|
# explicitly so the tunable stays visible rather than merely absent.
|
|
CTX_CHECKPOINTS=0
|
|
CACHE_RAM=8192
|
|
|
|
# Sampling defaults for requests that send none -- the values the vLLM
|
|
# predecessor served on this endpoint (`--override-generation-config`), so a
|
|
# client sees no change across the runtime swap. The model card's own
|
|
# recommendation is temp 1.0 / top_p 0.95; the harness sets its own per model in
|
|
# models.yml, so this is the floor, not the contract.
|
|
TEMPERATURE=0.7
|
|
TOP_P=0.95
|
|
TOP_K=20
|
|
MIN_P=0.0
|
|
PRESENCE_PENALTY=0.0
|
|
|
|
# Reasoning on, projected into reasoning_content (deepseek format) with no
|
|
# budget: the model decides how long to think.
|
|
REASONING=on
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|
|
|
|
# Chat template: empty = the one embedded in the GGUF (Qwen3.8's own).
|
|
CHAT_TEMPLATE_FILE=
|
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
|
|
|
# Speculative decoding: the model's OWN MTP head -- the blk.<n>.nextn.*
|
|
# tensors inside the target file. No sidecar file, no second copy of the
|
|
# weights: the draft context runs on the main model (shares_model), so the
|
|
# cost is one nextn layer's KV (4096 B/position) plus small buffers on the
|
|
# card that holds the last layers. This is what makes the 393216 pool fit:
|
|
# the DFlash2 sidecar's 1.92 GiB file and its buffers are off ROCm1.
|
|
# n_max: 3 is the measured MTP winner (see the table below -- n=7 accepts 33 %
|
|
# of 147, n=3 accepts 58 % of 72). The drafter rides ROCm1.
|
|
SPEC_TYPE=draft-mtp
|
|
SPEC_DRAFT_N_MAX=3
|
|
SPEC_DRAFT_NGL=999
|
|
SPEC_DRAFT_DEVICE=ROCm1
|
|
SPEC_DRAFT_TYPE_K=f16
|
|
SPEC_DRAFT_TYPE_V=f16
|
|
|
|
# MTP (the model's own head) vs the DFlash2 drafter -- MEASURED 2026-09-16
|
|
# (temperature 0, 1k prompt / 512 out; the autoregressive baseline and both
|
|
# MTP depths were run in one session, same binary, same prompts):
|
|
#
|
|
# none 17.7 tok/s card1 26.07 GB
|
|
# draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
|
|
# draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
|
|
# draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
|
|
#
|
|
# All four produced IDENTICAL greedy text, so verification is lossless
|
|
# whichever proposer runs. DFlash2 is the faster proposer (2.46x over
|
|
# autoregressive, 34 % over the best MTP); the operator switched to MTP on
|
|
# 2026-09-17 for the VRAM, not the speed: the MTP head adds ~3.3 GB to the
|
|
# card that holds it, against 2.06 GB for the drafter's file, and it is the
|
|
# only drafter that fits the 393216 pool. Do not switch this without
|
|
# re-measuring.
|
|
#
|
|
# THE TWO CANNOT BE CHAINED. `--spec-type draft-mtp,draft-dflash` dies at load:
|
|
# context type MTP requested but model doesn't contain MTP layers
|
|
# failed to create MTP context
|
|
# A chain has ONE draft context, and MTP builds against whatever model that
|
|
# context holds -- here the DFlash2 file, which has no MTP layers. MTP's head is
|
|
# only loaded when MTP IS the draft model ("creating MTP draft context against
|
|
# the target model ..."), so a chain of the two is a structural dead end, not a
|
|
# tuning question. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md.
|
|
|
|
# ngram-mod: the draftless n-gram matcher (CPU hash table in host RAM, no VRAM,
|
|
# no model file, no draft context). It joins the chain unconditionally --
|
|
# unlike MTP, which needs a draft context and therefore cannot share one with
|
|
# DFlash2. NOT CURRENTLY ACTIVE: SPEC_TYPE is draft-mtp alone, and the serve
|
|
# script only passes --spec-ngram-mod-* when the type contains "ngram". The
|
|
# window below is retained because the DFlash2 + ngram chain was measured free
|
|
# on prose and 2.6x on prompt-echo -- one SPEC_TYPE change brings it back.
|
|
#
|
|
# WINDOW MEASURED 2026-09-16 (chain at each window, decode tok/s):
|
|
#
|
|
# n_min/n_max prose 1k/512 prompt-echo 1.5k/400
|
|
# 48/64 43.9 200.6 <- deployed
|
|
# 16/32 36.0 161.0
|
|
# 8/16 36.2 113.8
|
|
#
|
|
# The long window is the right one, and not only because it wins on echo: a
|
|
# SHORT window makes the matcher propose on short fragments of ordinary prose,
|
|
# where the proposals are rejected and the verification still costs -- which is
|
|
# the "ngram hurts" result the fleet measured on other payloads twice before.
|
|
# With 48/64 the chain is byte-identical to DFlash2 alone on prose (same 119
|
|
# drafts / 49 accepted) and on the identity prompt (70/53).
|
|
#
|
|
# Do not shorten these without re-running scripts/bench-spec-ab.py: the
|
|
# no-penalty property is what makes the chain safe as a default.
|
|
SPEC_NGRAM_N_MATCH=24
|
|
SPEC_NGRAM_N_MIN=48
|
|
SPEC_NGRAM_N_MAX=64
|
|
|
|
# Vision enabled: llama-server autoloads mmproj-BF16.gguf from MODEL_DIR
|
|
# (888 MiB, lands on ROCm0 -- the fit above charges it there). frank-small
|
|
# (Dankclaw, RTX 3090) remains the fleet's dedicated image endpoint; this
|
|
# makes frank-medium multimodal too, which is why the seam flips from
|
|
# "text only" to "vision loads with the model".
|
|
NO_MMPROJ=0
|