# Tunables for frank-medium (ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP
# UD-Q8_K_XL + magnitudedev/Qwen3.8-27B-DSpark-GGUF Q8_0 drafter), mainline llama.cpp
# on Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201).  Since 2026-09-16 this
# is the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
# routing proxy are retired.
#
# Hardware: two R9700 (Navi 48, RDNA4, 32,624 MiB each), no XGMI, different
# IOMMU groups -- every cross-card byte goes over PCIe.  The Raphael iGPU
# (gfx1036/Radeon 610M) enumerates as HIP device 2 and is hidden below so
# nothing can drift onto shared system RAM.
#
# WHY LAYER SPLIT AND NOT TENSOR (operator call, 2026-09-16): tensor parallel
# needs a cross-card allreduce per layer, and this host cannot dispatch one.
# RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr refuses it
# on a device without PCIe atomics -- the second card sits behind the B650
# chipset bridges (AtomicOpsCap: Routing-, read-only), so every collective that
# touches it dies with hipErrorIllegalState.  Measured on this box 2026-09-15
# (llama-bench, pp4096 / tg128, f16 KV, fa on):
#
#   single R9700    1160.9 / 19.42
#   layer  2x       1991.2 / 18.76      (859.1 / 17.64 at depth 32768)
#   tensor 2x        271.8 / 26.26      (no RCCL: butterfly fallback, -7x prefill)
#
# Layer split distributes layers in order; only the activation at the boundary
# crosses the wire, once per forward pass.  That is the "pipeline parallelism"
# this service wanted, and it needs no collective at all.
#
# Runtime: mainline llama.cpp master fb27a525 (2026-09-16), gfx1201, built by
# scripts/build-llama-rdna4.sh against a private ROCm 7.14.0a20260624 toolchain
# that is bundled next to the binary (RPATH $ORIGIN/../lib, no LD_LIBRARY_PATH).
# The ref is pinned by TWO features: DFlash2 (ggml-org/llama.cpp#27342 -- the
# drafter is detected from its GGUF) and qwen35 MTP.  Do not float it.
#
# Model: ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL,
# 29.29 GiB, arch qwen35, Unsloth Dynamic 3.0 (per-tensor types from Unsloth's
# own quant of the same size, on Unsloth's importance matrix).  Abliteration is
# orcarouter's single refusal direction projected out of 131 tensors; the vision
# tower is untouched but unused here (text only, see NO_MMPROJ).  The file also
# carries the model's MTP head (blk.64.nextn.*), so a file-level switch to
# --spec-type draft-mtp is available without a second copy in VRAM.
#
# Drafter: magnitudedev/Qwen3.8-27B-DSpark-GGUF @ Q8_0, 1.27 GiB: 5 layers,
# hidden 5120, block-diffusion proposal with the Markov head (dspark).
# Decoding stays lossless -- the target verifies every drafted token. Its KV
# spans the whole context (see SIZING): unlike DFlash2 it declares no sliding
# window, and that is what the tensor split below pays for.
#
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
# per-sequence state is CONSTANT-SIZE, 16 full-attention layers at 4 KV heads x
# 256 head_dim.  A token therefore costs 2 x 4 x 256 x 2 B = 64 KiB of KV at f16
# in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet
# state at any depth.
#
# SIZING, measured (rocm-smi, 2026-09-16).  Weights split across the two cards;
# the drafter AND its KV ride the second.  The loader's compute buffers are part
# of what is seen below -- they are allocated at load, not per request:
#
#   CTX_SIZE  split  GPU 0      GPU 1      headroom on the busier card
#   131072    1,1    19.89 GB   24.30 GB   9.9 GB / 31 %   (DFlash2)
#   262144    1,1    24.72 GB   29.14 GB   5.07 GB / 15 %  (DFlash2) <- before
#   262144    38,27  28.52 GB   30.71 GB   3.50 GB / 10 %  (DSpark)  <- deployed
#
# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
# at 114688 was the vLLM predecessor's, not this runtime's.  If a future change
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
#
# WHY THE SPLIT IS 38:27.  The draft context is built with `n_ctx =
# llama_n_ctx(ctx_tgt)` (common/speculative.cpp), so the drafter's KV scales
# with the target window, and DSpark -- unlike DFlash2, which declared
# sliding_window 2048 and cost ~40 MiB -- caches all of it: 5 layers x 2 x 8 kv
# heads x 128 dim x 2 B = 20 KiB per position, i.e. 5.0 GiB at 262144.  At a
# 1:1 split that put ROCm1 at 33.9 GB of 34.2 GB and the draft's first graph
# capture died with `ROCm error: out of memory` (in
# common_speculative_impl_draft_dflash::process).  Handing three target layers
# to ROCm0 buys back ~3 GiB of headroom on both cards at the full window.
#
# EQUALITY WAS TRIED AND REJECTED.  39:26 levels the cards more closely (3.86 /
# 4.69 GiB free -- an 0.83 GiB gap, against 2.08 GiB here), but that server
# died on its eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults
# and the engine surfaced `ROCm error: an illegal memory access was
# encountered` -> ggml_abort, i.e. a fault in the cross-card boundary copy,
# which is this layer split's hot path.  38:27 has since served the whole e2e
# gate plus a burst of varied requests with none.  Do not chase an exact
# balance: take the skew that fits and leave the drafter's card on the safe
# side of the gap.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
# those bare vars; 2026-08-13 incident).  llama.cpp takes --host/--port/--temp
# as flags only.

# llama-server: mainline fb27a525 built for gfx1201 by scripts/build-llama-rdna4.sh.
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server

# Payload, fetched with scripts/download-frank-medium-gguf.sh.
MODEL_DIR=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
MODEL_FILE=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DSpark-GGUF
DRAFT_MODEL_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf

# Fleet identity.  7770 is the fleet's dankserver endpoint; the alias is the id
# at /v1/models and what every client asks for.
ALIAS=frank-medium
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Device placement.  HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2)
# so nothing can drift onto shared system RAM; DEVICES is what llama.cpp splits
# across, and neither of the two cards has an XGMI link to the other.
HIP_VISIBLE_DEVICES=0,1
DEVICES=ROCm0,ROCm1
SPLIT_MODE=layer

# Skewed 38:27 rather than 1:1: ROCm1 also carries the drafter and the whole of
# its KV, so it must hold fewer target layers for the cards to end level -- at
# 1:1 the drafter's card would sit ~9 GiB above the other.  This is the measured
# point that keeps BOTH cards above a 3 GiB floor at the full window; a tighter
# balance was tried and rejected (see SIZING above).
TENSOR_SPLIT=38,27

NGL=999
# Auto-fit has no business resizing a context we set explicitly.
FIT=off
FLASH_ATTN=on

# One slot at the model's full configured window (n_ctx_train 262144), which was
# measured to fit before it was deployed: see SIZING above.  --parallel DIVIDES
# --ctx-size between slots, so one slot carries the whole window and a second
# concurrent request queues behind it.
CTX_SIZE=262144
PARALLEL=1

BATCH_SIZE=2048
UBATCH_SIZE=512

# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
NUM_THREADS=8
THREAD_BATCH=8

# f16 K/V: an agentic checkpoint is the wrong place to spend KV precision.
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
# Context checkpoints: 0 is the build's default -- no per-slot KV snapshots, so
# a re-sent or rewound conversation re-prefills instead of rewinding.  Stated
# explicitly so the tunable stays visible rather than merely absent.
CTX_CHECKPOINTS=0
CACHE_RAM=8192

# Sampling defaults for requests that send none -- the values the vLLM
# predecessor served on this endpoint (`--override-generation-config`), so a
# client sees no change across the runtime swap.  The model card's own
# recommendation is temp 1.0 / top_p 0.95; the harness sets its own per model in
# models.yml, so this is the floor, not the contract.
TEMPERATURE=0.7
TOP_P=0.95
TOP_K=20
MIN_P=0.0
PRESENCE_PENALTY=0.0

# Reasoning on, projected into reasoning_content (deepseek format) with no
# budget: the model decides how long to think.
REASONING=on
REASONING_FORMAT=deepseek
REASONING_BUDGET=-1

# Chat template: empty = the one embedded in the GGUF (Qwen3.8's own).
CHAT_TEMPLATE_FILE=
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'

# Speculative decoding: the DSpark block-diffusion drafter PLUS the draftless
# n-gram matcher. DSpark's GGUF carries block_size 7, so at most 6 draft
# tokens per verification step (llama.cpp asserts n_max <= block_size - 1).
# The drafter rides ROCm1, the card that ends up holding the target's last
# layers.
SPEC_TYPE=draft-dspark,ngram-mod
SPEC_DRAFT_N_MAX=6
SPEC_DRAFT_NGL=999
SPEC_DRAFT_DEVICE=ROCm1
SPEC_DRAFT_TYPE_K=f16
SPEC_DRAFT_TYPE_V=f16

# MTP (the model's own head) vs the drafter -- MEASURED 2026-09-16 (temperature
# 0, 1k prompt / 512 out; the autoregressive baseline and both MTP depths were
# run against the deployed configuration in one session, same binary, same
# prompts):
#
#   none            17.7 tok/s                     card1 26.07 GB
#   draft-mtp  n=3  32.4 tok/s  58 % accepted       card1 29.35 GB
#   draft-mtp  n=7  26.7 tok/s  33 % accepted       card1 29.65 GB
#   draft-dflash 7  43.5 tok/s  41 % accepted       card1 29.42 GB
#
# All four produced IDENTICAL greedy text, so verification is lossless whichever
# proposer runs.  DFlash2 is worth 2.46x over autoregressive and 34 % over the
# best MTP; deeper MTP drafting costs more than it returns (n=7 accepts 33 % of
# 147 proposals, n=3 accepts 58 % of 72).  The MTP head is not free either:
# selecting it loads blk.64.nextn.* and adds ~3.3 GB to the card that holds it,
# against 2.06 GB for the drafter's file.  Do not switch this without
# re-measuring.
#
# THE TWO CANNOT BE CHAINED. `--spec-type draft-mtp,draft-dflash` dies at load:
#   context type MTP requested but model doesn't contain MTP layers
#   failed to create MTP context
# A chain has ONE draft context, and MTP builds against whatever model that
# context holds -- here the DFlash2 file, which has no MTP layers. MTP's head is
# only loaded when MTP IS the draft model ("creating MTP draft context against
# the target model ..."), so a chain of the two is a structural dead end, not a
# tuning question. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md.

# ngram-mod: the draftless n-gram matcher (CPU hash table in host RAM, no VRAM,
# no model file, no draft context). It joins the chain unconditionally --
# unlike MTP, which needs a draft context and therefore cannot share one with
# DFlash2. Verified on the loaded server: with ngram-mod as the ONLY spec type,
# a prose prompt produces ZERO drafts (plain autoregressive) while a
# prompt-echo prompt accepts 100 % of them, so its drafts are unmistakable.
#
# WINDOW MEASURED 2026-09-16 (chain at each window, decode tok/s):
#
#   n_min/n_max    prose 1k/512   prompt-echo 1.5k/400
#   48/64          43.9           200.6     <- deployed
#   16/32          36.0           161.0
#    8/16          36.2           113.8
#
# The long window is the right one, and not only because it wins on echo: a
# SHORT window makes the matcher propose on short fragments of ordinary prose,
# where the proposals are rejected and the verification still costs -- which is
# the "ngram hurts" result the fleet measured on other payloads twice before.
# With 48/64 the chain is byte-identical to DFlash2 alone on prose (same 119
# drafts / 49 accepted) and on the identity prompt (70/53).
#
# Do not shorten these without re-running scripts/bench-spec-ab.py: the
# no-penalty property is what makes the chain safe as a default.
SPEC_NGRAM_N_MATCH=24
SPEC_NGRAM_N_MIN=48
SPEC_NGRAM_N_MAX=64

# Text only: the server must not autoload a vision tower.  Vision stays on
# frank-small (Dankclaw, RTX 3090), which is the fleet's image endpoint.
NO_MMPROJ=1