Frank
d374ab963f
Operator call 2026-09-19: switch the target to orcarouter/Qwen3.8-27B-Uncensored-GGUF Q8_0 (gated repo; the fetch script now requires an HF token and verifies sha256, not just size), take the drafter to the Q4_K_M rung of z-lab/Qwen3.8-27B-DFlash2-GGUF, cut the pool to two 163840-token conversations (327680, entirely inside the trained 262144 window), and balance the tensors across the cards (32:33) in place of the 35:30 x16-lean. The in-file MTP head stays one SPEC_TYPE change away.
319 lines
17 KiB
Bash
319 lines
17 KiB
Bash
# Tunables for frank-medium (orcarouter/Qwen3.8-27B-Uncensored-GGUF Q8_0 +
|
|
# z-lab/Qwen3.8-27B-DFlash2-GGUF Q4_K_M drafter), mainline llama.cpp on
|
|
# Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this is
|
|
# the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
|
|
# routing proxy are retired.
|
|
#
|
|
# Hardware: two R9700 (Navi 48, RDNA4, 32,624 MiB each), no XGMI, different
|
|
# IOMMU groups -- every cross-card byte goes over PCIe. The Raphael iGPU
|
|
# (gfx1036/Radeon 610M) enumerates as HIP device 2 and is hidden below so
|
|
# nothing can drift onto shared system RAM.
|
|
#
|
|
# WHY LAYER SPLIT AND NOT TENSOR (operator call, 2026-09-16): tensor parallel
|
|
# needs a cross-card allreduce per layer, and this host cannot dispatch one.
|
|
# RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr refuses it
|
|
# on a device without PCIe atomics -- the second card sits behind the B650
|
|
# chipset bridges (AtomicOpsCap: Routing-, read-only), so every collective that
|
|
# touches it dies with hipErrorIllegalState. Measured on this box 2026-09-15
|
|
# (llama-bench, pp4096 / tg128, f16 KV, fa on):
|
|
#
|
|
# single R9700 1160.9 / 19.42
|
|
# layer 2x 1991.2 / 18.76 (859.1 / 17.64 at depth 32768)
|
|
# tensor 2x 271.8 / 26.26 (no RCCL: butterfly fallback, -7x prefill)
|
|
#
|
|
# Layer split distributes layers in order; only the activation at the boundary
|
|
# crosses the wire, once per forward pass. That is the "pipeline parallelism"
|
|
# this service wanted, and it needs no collective at all.
|
|
#
|
|
# Runtime: mainline llama.cpp master fb27a525 (2026-09-16), gfx1201, built by
|
|
# scripts/build-llama-rdna4.sh against a private ROCm 7.14.0a20260624 toolchain
|
|
# that is bundled next to the binary (RPATH $ORIGIN/../lib, no LD_LIBRARY_PATH).
|
|
# The ref is pinned by TWO features: DFlash2 (ggml-org/llama.cpp#27342 -- the
|
|
# drafter is detected from its GGUF) and qwen35 MTP. Do not float it.
|
|
#
|
|
# Model: orcarouter/Qwen3.8-27B-Uncensored-GGUF @ Q8_0, 27.0 GiB, arch qwen35
|
|
# (verified from the GGUF header 2026-09-19: 262144 trained window, 24 heads /
|
|
# 4 KV heads, nextn_predict_layers=1). The file ALSO carries the model's MTP
|
|
# head (blk.64.nextn.*), so --spec-type draft-mtp is available without a second
|
|
# copy in VRAM -- one SPEC_TYPE change, the drafter below stays on disk.
|
|
#
|
|
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q4_K_M, 1.06 GiB (operator call
|
|
# 2026-09-19: the Q4 rung -- the drafter's job is throughput, not output
|
|
# quality, and the target verifies every drafted token). 5 layers, hidden
|
|
# 5120, a block-diffusion proposal (block_size 8) under a 2048-token sliding
|
|
# window (dflash.attention.sliding_window=2048, verified from the header).
|
|
# The window is what makes it cheap HERE: its KV is bounded at 2048 positions
|
|
# (~40 MiB) instead of spanning the context (see SIZING).
|
|
#
|
|
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
|
|
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
|
|
# per-sequence state is CONSTANT-SIZE, 16 full-attention layers at 4 KV heads x
|
|
# 256 head_dim. A token therefore costs 2 x 4 x 256 x 2 B = 64 KiB of KV at f16
|
|
# in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet
|
|
# state at any depth.
|
|
#
|
|
# SIZING, the deployed shape (32:33 / ctx 327680 / 6144x512 / two slots /
|
|
# DFlash2 Q4_K_M), measured 2026-09-19 after the model swap (rocm-smi):
|
|
#
|
|
# card 0 (x16) 26.9 GiB used, 5.7 GiB free
|
|
# card 1 (x1) 27.8 GiB used, 4.8 GiB free
|
|
#
|
|
# The predecessor's measured table (rocm-smi, 2026-09-16, UD-Q8_K_XL target)
|
|
# stays as the calibration history:
|
|
#
|
|
# CTX split batch slots drafter GPU 0 free GPU 1 free note (ubatch 1536)
|
|
# 262144 1,1 2048 1 DFlash2 8.64 GiB 4.43 GiB day's start
|
|
# 262144 39,26 6144 2 DFlash2 2.42 GiB 7.20 GiB
|
|
# 262144 38,27 6144 2 DFlash2 3.85 GiB 5.77 GiB
|
|
# 262144 37,28 6144 2 DFlash2 4.28 GiB 5.34 GiB
|
|
# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB
|
|
# 262144 35,30 6144 2 DFlash2 5.14 GiB 4.48 GiB
|
|
# 262144 40,25 6144 2 DSpark 1.87 GiB 1.86 GiB (retired)
|
|
#
|
|
# ...and the same deployed shape re-measured at ubatch 512, after llama-bench
|
|
# showed 512 is the throughput optimum and costs 3.71 GiB less in total (see the
|
|
# batch block below):
|
|
#
|
|
# 262144 36,29 6144 512 2 DFlash2 6.01 GiB 7.32 GiB ubatch only
|
|
# 262144 35,30 6144 512 2 DFlash2 6.44 GiB 6.90 GiB
|
|
#
|
|
# ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE. This is a hybrid
|
|
# stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a
|
|
# DeltaNet layer shifts ~0.43 GiB while an attention layer shifts ~1.43 GiB
|
|
# (0.45 of weights plus ~1.0 GiB of its own f16 KV at 262144). Both step sizes
|
|
# were measured, which is why the balance is found by measurement rather than by
|
|
# interpolating between splits.
|
|
#
|
|
# WHY THE SPLIT IS 32:33 (operator call, 2026-09-19). The operator asked for
|
|
# the tensors BALANCED across the two cards, which for 65 layers is 32:33 --
|
|
# the most even split that exists. The predecessor's 35:30 skew existed to lean
|
|
# work onto the x16 card (ROCm0), because ROCm1's Gen4 x1 chipset path made it
|
|
# the 98 % utilization bottleneck; the measured trade of the balanced point is
|
|
# that the x1 card runs hotter while the x16 card idles a little more, and the
|
|
# operator accepted that for symmetry. Do not re-skew without asking.
|
|
#
|
|
# THE TWO CARDS' PATHS ARE NOT SYMMETRIC (measured, 2026-09-17): ROCm0
|
|
# (03:00.0) hangs off CPU root port 00:01.1 at Gen4 x16 end to end (~31.5 GB/s);
|
|
# ROCm1 (0b:00.0) rides the B650 chipset through bridges that each negotiate
|
|
# Gen4 x1 (~1.97 GB/s). The layer-split boundary itself is not the cost
|
|
# (~5.2 MB of activation per 512-token ubatch rides the wire); the x1 card is
|
|
# COMPUTE-bound, not PCIe-bound.
|
|
#
|
|
# A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth
|
|
# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
|
|
# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
|
|
# i.e. a fault in the cross-card boundary copy, this layer split's hot path. It
|
|
# has not reproduced since, but the signature is worth recognising -- it is NOT
|
|
# an OOM, and no amount of headroom prevents it.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
|
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
|
# as flags only.
|
|
|
|
# llama-server: mainline fb27a525 built for gfx1201 by scripts/build-llama-rdna4.sh.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server
|
|
|
|
# Payload, fetched with scripts/download-frank-medium-gguf.sh.
|
|
MODEL_DIR=/home/dank/models/Qwen3.8-27B-Uncensored-GGUF
|
|
MODEL_FILE=Qwen3.8-27B-Uncensored-Q8_0.gguf
|
|
# The f16 vision projector, same repo as the target. llama-server loads it via
|
|
# the explicit --mmproj flag (no autoload in this build); the serve script
|
|
# refuses to start without it when vision is enabled, so a missing file is a
|
|
# deploy error, not a text-only server that pretends to be multimodal.
|
|
MM_PROJ_FILE=mmproj-Qwen3.8-27B-Uncensored-f16.gguf
|
|
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF
|
|
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q4_K_M.gguf
|
|
|
|
# Fleet identity. 7770 is the fleet's dankserver endpoint; the alias is the id
|
|
# at /v1/models and what every client asks for.
|
|
ALIAS=frank-medium
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7770
|
|
# Device placement. HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2)
|
|
# so nothing can drift onto shared system RAM; DEVICES is what llama.cpp splits
|
|
# across, and neither of the two cards has an XGMI link to the other.
|
|
HIP_VISIBLE_DEVICES=0,1
|
|
DEVICES=ROCm0,ROCm1
|
|
SPLIT_MODE=layer
|
|
|
|
# 32:33 -- balanced (operator call 2026-09-19): 65 layers split as evenly as
|
|
# they can be, ROCm1 (the last-layers card, which also carries the drafter and
|
|
# the output layers) holding the extra one. See the WHY THE SPLIT IS 32:33
|
|
# block above for the trade against the predecessor's 35:30 x16-lean.
|
|
TENSOR_SPLIT=32,33
|
|
|
|
NGL=999
|
|
# Auto-fit has no business resizing a context we set explicitly.
|
|
FIT=off
|
|
FLASH_ATTN=on
|
|
|
|
# 327680 positions over TWO slots, so each conversation gets 163840 --
|
|
# --parallel DIVIDES --ctx-size, so more slots would shrink every
|
|
# conversation rather than grow the cache. 163840 is inside the checkpoint's
|
|
# trained 262144 window, so the whole pool is trained territory (no rope
|
|
# extrapolation, unlike the predecessor's 393216).
|
|
CTX_SIZE=327680
|
|
PARALLEL=2
|
|
|
|
# MEASURED 2026-09-16 with scripts/bench-batch-ubatch.sh on this box: llama-bench,
|
|
# a 64k prefill plus a 128-token decode with 64k of KV already resident, 3 reps,
|
|
# one invocation per cell, the deployed split and f16 KV. --ubatch-size is the
|
|
# physical batch the graph is built at -- and the only one of the two knobs that
|
|
# costs VRAM:
|
|
#
|
|
# batch ubatch prefill tok/s GPU 0 / GPU 1 free
|
|
# 6144 1536 1164.96 +- 2.30 10.28 / 11.71 GiB
|
|
# 6144 768 1135.03 +- 1.09 11.29 / 13.28 GiB
|
|
# 6144 512 1192.93 +- 0.91 11.63 / 13.81 GiB <- deployed
|
|
# 6144 384 1068.04 +- 0.79 11.80 / 14.07 GiB
|
|
# 6144 256 1045.98 +- 1.69 11.97 / 14.33 GiB
|
|
# 4096 1536 1161.90 +- 0.96 10.28 / 11.71 GiB
|
|
# 2048 1536 1170.23 +- 0.86 10.28 / 11.71 GiB
|
|
# 2048 512 1193.88 +- 1.01 11.63 / 13.81 GiB
|
|
#
|
|
# THE PLATEAU STARTS AT 512. 512 -> 1536 buys ~2 % of prefill -- inside the
|
|
# run-to-run spread, and the 512 cells were the faster pair in BOTH batch
|
|
# columns -- while costing 1.35 GiB on ROCm0 and 2.10 GiB on ROCm1, so the
|
|
# deployment takes the cheap end. BELOW 512 the SMs are no longer saturated:
|
|
# 384 and 256 fall ~11 % and ~12 %. The logical batch is free either way --
|
|
# 2048, 4096 and 6144 land within 2 % of each other at a fixed ubatch and use
|
|
# the SAME VRAM, because the compute buffers are sized by ubatch alone.
|
|
#
|
|
# The prefill figure is a property of the SHAPE, not of this knob: the same
|
|
# sweep with no depth in front of it (depth 0) runs 1543 tok/s at ubatch 256 and
|
|
# 1723 at 1536 -- the plateau is in the same place, and the ~32 % between the two
|
|
# tables is what attending over 64k of resident KV costs. Decode is 15.88 tok/s
|
|
# at depth 65536 on EVERY ubatch (17.80 at depth 0), so this knob cannot touch
|
|
# generation at all.
|
|
#
|
|
# 8192/2048 stays untested under this drafter (the "2020 MiB pp buffer fails on
|
|
# ROCm1" note was measured under DSpark, when it had 0.08 GiB free), and there is
|
|
# no longer a reason to want it: batch buys nothing measurable here.
|
|
BATCH_SIZE=6144
|
|
UBATCH_SIZE=512
|
|
|
|
# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
|
|
# f16 K/V: an agentic checkpoint is the wrong place to spend KV precision.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
# Context checkpoints: 0 -- no per-slot KV snapshots at all. The build's
|
|
# default is 32 (this binary's --help), so the 0 below is a deliberate
|
|
# deviation, not an inherited default: a checkpoint is a VRAM->host-RAM copy
|
|
# of part of a slot's KV, taken so a partially-matching prompt can be restored
|
|
# instead of re-prefilled. This endpoint trades that away -- its reuse comes
|
|
# from the slot prompt and from the RAM-side prompt cache below. Stated
|
|
# explicitly so the tunable stays visible rather than merely absent.
|
|
CTX_CHECKPOINTS=0
|
|
# PROMPT CACHE, HOST RAM -- this is the one cache tunable that is NOT VRAM.
|
|
# --cache-ram caps the total bytes of saved prompt states
|
|
# (server_prompt_cache, tools/server/server-task.cpp): update() evicts
|
|
# oldest-first past the cap, and alloc() REFUSES outright any single state
|
|
# bigger than the whole cap. A state is the serialized KV of a whole sequence
|
|
# plus its DeltaNet state plus the draft context's own KV, so at this shape it
|
|
# costs ~68 KiB per token --
|
|
# 16 full-attention layers x 4 kv heads x 256 dim x 2 B x 2 (K,V) = 64 KiB
|
|
# plus the draft context's window-bounded KV (a ~40 MiB constant for DFlash2)
|
|
# -- which makes the cap a TOKEN budget in disguise: 16384 MiB holds ~240k
|
|
# tokens, i.e. a FULL 163840-token slot (~10.6 GiB) with ~5 GiB of the budget
|
|
# spare for a second entry.
|
|
#
|
|
# The cost is host RAM, allocated on demand and never pinned (plain heap
|
|
# vectors), so no VRAM number above moves. dankserver has 62 GiB beside the
|
|
# media-stack co-tenants, so the cache is bounded rather than unlimited (-1):
|
|
# the seam test refuses a budget over half the host's RAM.
|
|
CACHE_RAM=16384
|
|
|
|
# Sampling defaults for requests that send none -- the values the vLLM
|
|
# predecessor served on this endpoint (`--override-generation-config`), so a
|
|
# client sees no change across the runtime swap. The model card's own
|
|
# recommendation is temp 1.0 / top_p 0.95; the harness sets its own per model in
|
|
# models.yml, so this is the floor, not the contract.
|
|
TEMPERATURE=0.7
|
|
TOP_P=0.95
|
|
TOP_K=20
|
|
MIN_P=0.0
|
|
PRESENCE_PENALTY=0.0
|
|
|
|
# Reasoning on, projected into reasoning_content (deepseek format) with no
|
|
# budget: the model decides how long to think.
|
|
REASONING=on
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|
|
|
|
# Chat template: empty = the one embedded in the GGUF (Qwen3.8's own).
|
|
CHAT_TEMPLATE_FILE=
|
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
|
|
|
# Speculative decoding: the DFlash2 sidecar (operator call 2026-09-19). The
|
|
# target file also carries its own MTP head (nextn_predict_layers=1), so
|
|
# draft-mtp is one SPEC_TYPE change away -- the drafter file stays on disk.
|
|
# n_max 7 is the measured DFlash2 winner (see the table below); the drafter
|
|
# rides ROCm1, the card that holds the last layers.
|
|
SPEC_TYPE=draft-dflash
|
|
SPEC_DRAFT_N_MAX=7
|
|
SPEC_DRAFT_NGL=999
|
|
SPEC_DRAFT_DEVICE=ROCm1
|
|
SPEC_DRAFT_TYPE_K=f16
|
|
SPEC_DRAFT_TYPE_V=f16
|
|
|
|
# DFlash2 vs the model's own MTP head -- MEASURED 2026-09-16 on the predecessor
|
|
# payload (temperature 0, 1k prompt / 512 out; the autoregressive baseline and
|
|
# both MTP depths were run in one session, same binary, same prompts):
|
|
#
|
|
# none 17.7 tok/s card1 26.07 GB
|
|
# draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
|
|
# draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
|
|
# draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
|
|
#
|
|
# All four produced IDENTICAL greedy text, so verification is lossless
|
|
# whichever proposer runs. DFlash2 is the faster proposer (2.46x over
|
|
# autoregressive, 34 % over the best MTP). The MTP head adds ~3.3 GB to the
|
|
# card that holds it, against 1.06 GB for the Q4_K_M drafter's file -- at the
|
|
# 327680 pool the drafter's window-bounded KV (~40 MiB) leaves headroom on
|
|
# both cards, so the speed wins.
|
|
#
|
|
# THE TWO CANNOT BE CHAINED. `--spec-type draft-mtp,draft-dflash` dies at load:
|
|
# context type MTP requested but model doesn't contain MTP layers
|
|
# failed to create MTP context
|
|
# A chain has ONE draft context, and MTP builds against whatever model that
|
|
# context holds -- here the DFlash2 file, which has no MTP layers. MTP's head is
|
|
# only loaded when MTP IS the draft model ("creating MTP draft context against
|
|
# the target model ..."), so a chain of the two is a structural dead end, not a
|
|
# tuning question. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md.
|
|
|
|
# ngram-mod: the draftless n-gram matcher (CPU hash table in host RAM, no VRAM,
|
|
# no model file, no draft context). It joins the chain unconditionally --
|
|
# unlike MTP, which needs a draft context and therefore cannot share one with
|
|
# DFlash2. NOT CURRENTLY ACTIVE: SPEC_TYPE is draft-dflash alone, and the serve
|
|
# script only passes --spec-ngram-mod-* when the type contains "ngram". The
|
|
# window below is retained because the DFlash2 + ngram chain was measured free
|
|
# on prose and 2.6x on prompt-echo -- one SPEC_TYPE change brings it back.
|
|
#
|
|
# WINDOW MEASURED 2026-09-16 (chain at each window, decode tok/s):
|
|
#
|
|
# n_min/n_max prose 1k/512 prompt-echo 1.5k/400
|
|
# 48/64 43.9 200.6
|
|
# 16/32 36.0 161.0
|
|
# 8/16 36.2 113.8
|
|
#
|
|
# The long window is the right one, and not only because it wins on echo: a
|
|
# SHORT window makes the matcher propose on short fragments of ordinary prose,
|
|
# where the proposals are rejected and the verification still costs -- which is
|
|
# the "ngram hurts" result the fleet measured on other payloads twice before.
|
|
# With 48/64 the chain is byte-identical to DFlash2 alone on prose (same 119
|
|
# drafts / 49 accepted) and on the identity prompt (70/53).
|
|
#
|
|
# Do not shorten these without re-running scripts/bench-spec-ab.py: the
|
|
# no-penalty property is what makes the chain safe as a default.
|
|
SPEC_NGRAM_N_MATCH=24
|
|
SPEC_NGRAM_N_MIN=48
|
|
SPEC_NGRAM_N_MAX=64
|
|
|
|
# Vision enabled: llama-server loads mmproj-Qwen3.8-27B-Uncensored-f16.gguf
|
|
# (931 MiB, lands on ROCm0 -- the fit charges it there) via the explicit
|
|
# --mmproj flag. This keeps frank-medium multimodal across the model swap.
|
|
NO_MMPROJ=0
|