serverAppConfigs/scripts/frank-medium.env

# Tunables for frank-medium (ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP
# UD-Q8_K_XL + z-lab/Qwen3.8-27B-DFlash2-GGUF Q8_0 drafter), mainline llama.cpp
# on Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201).  Since 2026-09-16 this
# is the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
# routing proxy are retired.
#
# Hardware: two R9700 (Navi 48, RDNA4, 32,624 MiB each), no XGMI, different
# IOMMU groups -- every cross-card byte goes over PCIe.  The Raphael iGPU
# (gfx1036/Radeon 610M) enumerates as HIP device 2 and is hidden below so
# nothing can drift onto shared system RAM.
#
# WHY LAYER SPLIT AND NOT TENSOR (operator call, 2026-09-16): tensor parallel
# needs a cross-card allreduce per layer, and this host cannot dispatch one.
# RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr refuses it
# on a device without PCIe atomics -- the second card sits behind the B650
# chipset bridges (AtomicOpsCap: Routing-, read-only), so every collective that
# touches it dies with hipErrorIllegalState.  Measured on this box 2026-09-15
# (llama-bench, pp4096 / tg128, f16 KV, fa on):
#
#   single R9700    1160.9 / 19.42
#   layer  2x       1991.2 / 18.76      (859.1 / 17.64 at depth 32768)
#   tensor 2x        271.8 / 26.26      (no RCCL: butterfly fallback, -7x prefill)
#
# Layer split distributes layers in order; only the activation at the boundary
# crosses the wire, once per forward pass.  That is the "pipeline parallelism"
# this service wanted, and it needs no collective at all.
#
# Runtime: mainline llama.cpp master fb27a525 (2026-09-16), gfx1201, built by
# scripts/build-llama-rdna4.sh against a private ROCm 7.14.0a20260624 toolchain
# that is bundled next to the binary (RPATH $ORIGIN/../lib, no LD_LIBRARY_PATH).
# The ref is pinned by TWO features: DFlash2 (ggml-org/llama.cpp#27342 -- the
# drafter is detected from its GGUF) and qwen35 MTP.  Do not float it.
#
# Model: ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL,
# 29.29 GiB, arch qwen35, Unsloth Dynamic 3.0 (per-tensor types from Unsloth's
# own quant of the same size, on Unsloth's importance matrix).  Abliteration is
# orcarouter's single refusal direction projected out of 131 tensors; the vision
# tower loads from mmproj-BF16.gguf (see NO_MMPROJ).  The file also
# carries the model's MTP head (blk.64.nextn.*), so a file-level switch to
# --spec-type draft-mtp is available without a second copy in VRAM.
#
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q8_0, 1.92 GiB: 5 layers, hidden
# 5120, a block-diffusion proposal (block_size 8) under a 2048-token sliding
# window.  Decoding stays lossless -- the target verifies every drafted token.
# The window is what makes it cheap HERE: its KV is bounded at 2048 positions
# (~40 MiB) instead of spanning the context (see SIZING).
#
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
# per-sequence state is CONSTANT-SIZE, 16 full-attention layers at 4 KV heads x
# 256 head_dim.  A token therefore costs 2 x 4 x 256 x 2 B = 64 KiB of KV at f16
# in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet
# state at any depth.
#
# SIZING, measured (rocm-smi, 2026-09-16).  Weights split across the two cards;
# the drafter AND its KV ride the second.  The loader's compute buffers are part
# of what is seen below -- they are allocated at load, not per request, and they
# scale with --ubatch-size:
#
#   CTX     split  batch  slots  drafter  GPU 0 free  GPU 1 free   note (ubatch 1536)
#   262144   1,1   2048   1      DFlash2   8.64 GiB    4.43 GiB   day's start
#   262144  39,26  6144   2      DFlash2   2.42 GiB    7.20 GiB
#   262144  38,27  6144   2      DFlash2   3.85 GiB    5.77 GiB
#   262144  37,28  6144   2      DFlash2   4.28 GiB    5.34 GiB
#   262144  36,29  6144   2      DFlash2   4.71 GiB    4.91 GiB
#   262144  35,30  6144   2      DFlash2   5.14 GiB    4.48 GiB
#   262144  40,25  6144   2      DSpark    1.87 GiB    1.86 GiB   (retired)
#
# ...and the same deployed shape re-measured at ubatch 512, after llama-bench
# showed 512 is the throughput optimum and costs 3.71 GiB less in total (see the
# batch block below):
#
#   262144  36,29  6144   512    2      DFlash2   6.01 GiB    7.32 GiB   ubatch only
#   262144  35,30  6144   512    2      DFlash2   6.44 GiB    6.90 GiB   <- deployed
#
# ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE.  This is a hybrid
# stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a
# DeltaNet layer shifts ~0.43 GiB while an attention layer shifts ~1.43 GiB
# (0.45 of weights plus ~1.0 GiB of its own f16 KV at 262144).  Both step sizes
# were measured, which is why the balance is found by measurement rather than by
# interpolating between splits.
#
# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
# at 114688 was the vLLM predecessor's, not this runtime's.  If a future change
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
#
# WHY THE SPLIT IS 35:30.  Three effects decide it, and all three are measured:
#   * THE DRAFTER AND ITS OUTPUT LAYERS RIDE ROCm1.  It carries the drafter's
#     weights, the draft graph and the last target layers, so it takes fewer
#     than half the layer count.
#   * THE UBATCH BUFFERS COST ROCm1 MORE THAN ROCm0, and shrinking them pays
#     ROCm1 back more: 1536 -> 512 freed 1.30 GiB on ROCm0 but 2.41 GiB on
#     ROCm1 at this shape.  Read the other way, that is exactly why the smaller
#     batch moved the balance toward ROCm1 -- and cost ROCm0 one layer.
#   * THE LAYER GRANULARITY IS THE FLOOR ON HOW WELL ANY SPLIT CAN BALANCE.  One
#     -ts step is one layer -- ~0.43 GiB DeltaNet, ~1.43 GiB attention -- so the
#     reachable minimum gap is comparable to one step rather than to zero.  35:30
#     sits at 0.46 GiB, i.e. one DeltaNet layer's worth, against 1.31 GiB at
#     36:29.
# It also leaves the most room: 6.44 / 6.90 GiB free clears the operator's 1.5
# GiB-per-card floor by ~5 GiB, so no headroom has to be bought out of CTX_SIZE
# and the full window stays.
#
# A FAULT TO WATCH.  A 39:26 split, at 1x batch / 1 slot, died on its eighth
# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
# i.e. a fault in the cross-card boundary copy, this layer split's hot path.  It
# has not reproduced (36:29 served this file's verification battery and the e2e
# gate, and 35:30 is the deployed shape), but the signature is worth recognising
# -- it is NOT an OOM, and no amount of headroom prevents it.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
# those bare vars; 2026-08-13 incident).  llama.cpp takes --host/--port/--temp
# as flags only.

# llama-server: mainline fb27a525 built for gfx1201 by scripts/build-llama-rdna4.sh.
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server

# Payload, fetched with scripts/download-frank-medium-gguf.sh.
MODEL_DIR=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
MODEL_FILE=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
# The BF16 vision projector, same repo as the target.  llama-server autoloads
# any mmproj*.gguf beside the model; the serve script refuses to start without
# it when vision is enabled, so a missing file is a deploy error, not a
# text-only server that pretends to be multimodal.
MM_PROJ_FILE=mmproj-BF16.gguf
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf

# Fleet identity.  7770 is the fleet's dankserver endpoint; the alias is the id
# at /v1/models and what every client asks for.
ALIAS=frank-medium
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Device placement.  HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2)
# so nothing can drift onto shared system RAM; DEVICES is what llama.cpp splits
# across, and neither of the two cards has an XGMI link to the other.
HIP_VISIBLE_DEVICES=0,1
DEVICES=ROCm0,ROCm1
SPLIT_MODE=layer

# 35:30 -- operator call 2026-09-17: lean the work onto the x16 card.  The
# two R9700s do NOT have symmetric PCIe paths.  ROCm0 (03:00.0) hangs off
# CPU root port 00:01.1 at Gen4 x16 end to end (~31.5 GB/s); ROCm1
# (0b:00.0) rides the B650 chipset through bridges that each negotiate
# Gen4 x1 (~1.97 GB/s).  Live rocm-smi showed the x1 card at 98% busy
# against 40% on the x16 card: the MTP drafter, the output layers and the
# heavier compute buffers are structurally pinned to the card holding the
# LAST layers (the draft context shares the target's weights there), so
# the bottlenecked card was doing most of the work.  Moving one layer to
# ROCm0 shifts ~0.43 GiB of DeltaNet weights (layer 34 is a DeltaNet --
# the 4k..4k+2 position of block 8) off the x1 card and gives the healthy
# card the bigger share.  The layer-split boundary itself is not the cost
# (~5.2 MB of activation per 512-token ubatch rides the wire), but the
# card behind the x1 link should not be the saturated one while its
# x16-sibling idles.
#
# WHY NOT FURTHER.  One -ts step is one LAYER: ~0.43 GiB (DeltaNet) or
# ~1.43 GiB (attention).  The next step, 36:29, would cross the
# full-attention layer at 35: the seam fit puts card 0 at 31.41 GiB
# (free 0.45 GiB, under the 1.0 GiB floor) with a 2.22 GiB gap (over the
# 1.483 GiB slack) -- both gates fail.  35:30 (fit: 30.59 / 30.01, gap
# 0.58) is the farthest safe shift toward the healthy card at this pool.
# Do not move it without re-measuring.

# MEASURED 2026-09-18 (isolated :7790, 2k-token decode, AVG rocm-smi util over
# the request -- peak util lies, a layer split touches both cards every token):
#
#   split   draft_dev  decode   AVG util0(x16)  AVG util1(x1)
#   35,30   ROCm1      31.52    41 %            98 %           <- deployed
#   36,29   ROCm1      31.49    43 %            98 %
#   35,30   ROCm0      31.52    42 %            98 %
#   36,29   ROCm0      31.48    43 %            98 %
#
# The x1 card stays the 98 % bottleneck in EVERY configuration.  The lever that
# was tried is moving the MTP drafter to the x16 card (SPEC_DRAFT_DEVICE=ROCm0):
# it does NOT flip the balance, because the drafter is a small nextn head and the
# per-step cost is dominated by the VERIFY phase -- both cards run the full target
# -- plus the LOGITS, which are pinned to the card holding the last layers.  A
# target-layer move (36:29) buys 2 points on the x16 card and erodes its VRAM
# margin (fit: free 0.45 GiB) for nothing.  The x1 card is COMPUTE-bound, not
# PCIe-bound (~5.2 MB of activation per 512-token ubatch crosses the link);
# saturating it costs no tokens beyond what any R9700 would.  35:30 is the
# balance; do not chase the x16 card past it.
TENSOR_SPLIT=35,30

NGL=999
# Auto-fit has no business resizing a context we set explicitly.
FIT=off
FLASH_ATTN=on

# 393216 positions over TWO slots, so each conversation gets 196608 --
# --parallel DIVIDES --ctx-size, so more slots would shrink every
# conversation rather than grow the cache.  The checkpoint's trained window is
# 262144 and the GGUF carries NO rope scaling (freq_base 1e7, no YaRN keys),
# so positions beyond 262144 extrapolate on the base rope: it runs, and 1.5x
# is a modest ask of a rope, but the top 131k of the pool is untrained
# territory.
CTX_SIZE=393216
PARALLEL=2

# MEASURED 2026-09-16 with scripts/bench-batch-ubatch.sh on this box: llama-bench,
# a 64k prefill plus a 128-token decode with 64k of KV already resident, 3 reps,
# one invocation per cell, the deployed split and f16 KV.  --ubatch-size is the
# physical batch the graph is built at -- and the only one of the two knobs that
# costs VRAM:
#
#   batch  ubatch   prefill tok/s        GPU 0 / GPU 1 free
#   6144   1536    1164.96 +- 2.30      10.28 / 11.71 GiB
#   6144    768    1135.03 +- 1.09      11.29 / 13.28 GiB
#   6144    512    1192.93 +- 0.91      11.63 / 13.81 GiB   <- deployed
#   6144    384    1068.04 +- 0.79      11.80 / 14.07 GiB
#   6144    256    1045.98 +- 1.69      11.97 / 14.33 GiB
#   4096   1536    1161.90 +- 0.96      10.28 / 11.71 GiB
#   2048   1536    1170.23 +- 0.86      10.28 / 11.71 GiB
#   2048    512    1193.88 +- 1.01      11.63 / 13.81 GiB
#
# THE PLATEAU STARTS AT 512.  512 -> 1536 buys ~2 % of prefill -- inside the
# run-to-run spread, and the 512 cells were the faster pair in BOTH batch
# columns -- while costing 1.35 GiB on ROCm0 and 2.10 GiB on ROCm1, so the
# deployment takes the cheap end.  BELOW 512 the SMs are no longer saturated:
# 384 and 256 fall ~11 % and ~12 %.  The logical batch is free either way --
# 2048, 4096 and 6144 land within 2 % of each other at a fixed ubatch and use
# the SAME VRAM, because the compute buffers are sized by ubatch alone.
#
# The prefill figure is a property of the SHAPE, not of this knob: the same
# sweep with no depth in front of it (depth 0) runs 1543 tok/s at ubatch 256 and
# 1723 at 1536 -- the plateau is in the same place, and the ~32 % between the two
# tables is what attending over 64k of resident KV costs.  Decode is 15.88 tok/s
# at depth 65536 on EVERY ubatch (17.80 at depth 0), so this knob cannot touch
# generation at all.
#
# 8192/2048 stays untested under this drafter (the "2020 MiB pp buffer fails on
# ROCm1" note was measured under DSpark, when it had 0.08 GiB free), and there is
# no longer a reason to want it: batch buys nothing measurable here.
BATCH_SIZE=6144
UBATCH_SIZE=512

# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
NUM_THREADS=8
THREAD_BATCH=8

# f16 K/V: an agentic checkpoint is the wrong place to spend KV precision.
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
# Context checkpoints: 0 -- no per-slot KV snapshots at all.  The build's
# default is 32 (this binary's --help), so the 0 below is a deliberate
# deviation, not an inherited default: a checkpoint is a VRAM->host-RAM copy
# of part of a slot's KV, taken so a partially-matching prompt can be restored
# instead of re-prefilled.  This endpoint trades that away -- its reuse comes
# from the slot prompt and from the RAM-side prompt cache below.  Stated
# explicitly so the tunable stays visible rather than merely absent.
CTX_CHECKPOINTS=0
# PROMPT CACHE, HOST RAM -- this is the one cache tunable that is NOT VRAM.
# --cache-ram caps the total bytes of saved prompt states
# (server_prompt_cache, tools/server/server-task.cpp): update() evicts
# oldest-first past the cap, and alloc() REFUSES outright any single state
# bigger than the whole cap.  A state is the serialized KV of a whole sequence
# plus its DeltaNet state plus the MTP draft context's own KV, so at this shape
# it costs 68 KiB per token --
#   16 full-attention layers x 4 kv heads x 256 dim x 2 B x 2 (K,V) = 64 KiB
#   plus 4096 B for the draft context (one nextn layer)
# -- which makes the cap a TOKEN budget in disguise: 8192 MiB held ~120k
# tokens, 16384 MiB holds ~240k.
#
# DOUBLED 8192 -> 16384 on 2026-09-18 because the old cap was BINDING, not
# theoretical: alloc() refused 137 states in the two days from 2026-09-16 with
#   - prompt state size 11388.990 MiB exceeds cache size limit 8192.000 MiB, skipping
# i.e. 11.0-11.4 GiB states -- conversations of ~170k tokens at the law above --
# so every deep conversation was silently uncacheable and re-prefilled from
# scratch on its next turn.  At 16384 a FULL 196608-token slot (~12.8 GiB) fits
# with ~3 GiB of the budget spare for a second entry.
#
# The cost is host RAM, allocated on demand and never pinned (plain heap
# vectors), so no VRAM number above moves.  dankserver has 62 GiB beside the
# media-stack co-tenants, so the cache is bounded rather than unlimited (-1):
# the seam test refuses a budget over half the host's RAM.
CACHE_RAM=16384

# Sampling defaults for requests that send none -- the values the vLLM
# predecessor served on this endpoint (`--override-generation-config`), so a
# client sees no change across the runtime swap.  The model card's own
# recommendation is temp 1.0 / top_p 0.95; the harness sets its own per model in
# models.yml, so this is the floor, not the contract.
TEMPERATURE=0.7
TOP_P=0.95
TOP_K=20
MIN_P=0.0
PRESENCE_PENALTY=0.0

# Reasoning on, projected into reasoning_content (deepseek format) with no
# budget: the model decides how long to think.
REASONING=on
REASONING_FORMAT=deepseek
REASONING_BUDGET=-1

# Chat template: empty = the one embedded in the GGUF (Qwen3.8's own).
CHAT_TEMPLATE_FILE=
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'

# Speculative decoding: the model's OWN MTP head -- the blk.<n>.nextn.*
# tensors inside the target file.  No sidecar file, no second copy of the
# weights: the draft context runs on the main model (shares_model), so the
# cost is one nextn layer's KV (4096 B/position) plus small buffers on the
# card that holds the last layers.  This is what makes the 393216 pool fit:
# the DFlash2 sidecar's 1.92 GiB file and its buffers are off ROCm1.
# n_max: 3 is the measured MTP winner (see the table below -- n=7 accepts 33 %
# of 147, n=3 accepts 58 % of 72).  The drafter rides ROCm1.
SPEC_TYPE=draft-mtp
SPEC_DRAFT_N_MAX=3
SPEC_DRAFT_NGL=999
SPEC_DRAFT_DEVICE=ROCm1
SPEC_DRAFT_TYPE_K=f16
SPEC_DRAFT_TYPE_V=f16

# MTP (the model's own head) vs the DFlash2 drafter -- MEASURED 2026-09-16
# (temperature 0, 1k prompt / 512 out; the autoregressive baseline and both
# MTP depths were run in one session, same binary, same prompts):
#
#   none            17.7 tok/s                     card1 26.07 GB
#   draft-mtp  n=3  32.4 tok/s  58 % accepted       card1 29.35 GB
#   draft-mtp  n=7  26.7 tok/s  33 % accepted       card1 29.65 GB
#   draft-dflash 7  43.5 tok/s  41 % accepted       card1 29.42 GB
#
# All four produced IDENTICAL greedy text, so verification is lossless
# whichever proposer runs.  DFlash2 is the faster proposer (2.46x over
# autoregressive, 34 % over the best MTP); the operator switched to MTP on
# 2026-09-17 for the VRAM, not the speed: the MTP head adds ~3.3 GB to the
# card that holds it, against 2.06 GB for the drafter's file, and it is the
# only drafter that fits the 393216 pool.  Do not switch this without
# re-measuring.
#
# THE TWO CANNOT BE CHAINED. `--spec-type draft-mtp,draft-dflash` dies at load:
#   context type MTP requested but model doesn't contain MTP layers
#   failed to create MTP context
# A chain has ONE draft context, and MTP builds against whatever model that
# context holds -- here the DFlash2 file, which has no MTP layers. MTP's head is
# only loaded when MTP IS the draft model ("creating MTP draft context against
# the target model ..."), so a chain of the two is a structural dead end, not a
# tuning question. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md.

# ngram-mod: the draftless n-gram matcher (CPU hash table in host RAM, no VRAM,
# no model file, no draft context).  It joins the chain unconditionally --
# unlike MTP, which needs a draft context and therefore cannot share one with
# DFlash2.  NOT CURRENTLY ACTIVE: SPEC_TYPE is draft-mtp alone, and the serve
# script only passes --spec-ngram-mod-* when the type contains "ngram".  The
# window below is retained because the DFlash2 + ngram chain was measured free
# on prose and 2.6x on prompt-echo -- one SPEC_TYPE change brings it back.
#
# WINDOW MEASURED 2026-09-16 (chain at each window, decode tok/s):
#
#   n_min/n_max    prose 1k/512   prompt-echo 1.5k/400
#   48/64          43.9           200.6     <- deployed
#   16/32          36.0           161.0
#    8/16          36.2           113.8
#
# The long window is the right one, and not only because it wins on echo: a
# SHORT window makes the matcher propose on short fragments of ordinary prose,
# where the proposals are rejected and the verification still costs -- which is
# the "ngram hurts" result the fleet measured on other payloads twice before.
# With 48/64 the chain is byte-identical to DFlash2 alone on prose (same 119
# drafts / 49 accepted) and on the identity prompt (70/53).
#
# Do not shorten these without re-running scripts/bench-spec-ab.py: the
# no-penalty property is what makes the chain safe as a default.
SPEC_NGRAM_N_MATCH=24
SPEC_NGRAM_N_MIN=48
SPEC_NGRAM_N_MAX=64

# Vision enabled: llama-server autoloads mmproj-BF16.gguf from MODEL_DIR
# (888 MiB, lands on ROCm0 -- the fit above charges it there).  frank-small
# (Dankclaw, RTX 3090) remains the fleet's dedicated image endpoint; this
# makes frank-medium multimodal too, which is why the seam flips from
# "text only" to "vision loads with the model".
NO_MMPROJ=0