# frank-large — ROCmFPX llama.cpp serving
# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF (Q8_0_ROCMFPX_AGENT, ftype
# 115) + its Q4_0 MTP draft head, on the Strix Halo iGPU (gfx1151), :7770.
#
# 2026-10-02 (Frank): "the highest quality of that repo, one full context,
# whatever runtime it requires". The highest-quality rung is Q8_0_ROCMFPX_AGENT
# (26.28 GiB, 8.39 bpw: 187 tensors at plain Q8_0 + 311 at
# GGML_TYPE_Q8_0_ROCMFPX); the AGENT routing is also the variant the card
# tells you to take with MTP, because it keeps the output-head tensors at full
# Q8_0 (187 vs 2), which is what the draft scores against. The runtime the
# format REQUIRES is the ROCmFPX fork — type 103 exists nowhere else.
#
# The shape: one session at the model's full native window (262,144), which is
# the whole point — the Q8_0 head costs quality the fleet wanted back, and
# the draft head pays for the bandwidth the bigger file spends.
#
# Budget (from the GGUF header, not from a boot log): attention KV lives only
# in the 16 full-attention layers (4 KV heads x 256 head_dim), 64 KiB/token
# with f16 K and V => 16.0 GiB at 262,144. The 48 Gated DeltaNet layers hold
# ~144 MiB of fixed state per sequence. Weights 26.28 + draft head 1.57 +
# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 45-47 GiB against the
# iGPU's ~114 GiB GTT aperture, so f16 KV (the unquantized choice) is affordable
# here in a way it is not on frank-small's 24 GiB card.

# The qualified ROCmFPX fork build (scripts/build-rocmfpx.sh pins the commit).
LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
# One directory per HF repo, created by scripts/download-frank-large-kingjones.sh.
MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
# The draft head: Q4_0, NOT the Q8_0 head. The head's per-step cost is its
# projection over a 248K vocabulary, so the cheaper head is faster end to end
# at equal-or-better acceptance (card: +6.0% at n-max 1, +10.2% at n-max 2).
DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
# Vision tower, separate from the LLM file. Empty string = text-only.
MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf

# Fleet endpoint: same port and alias as every frank-large before it, so the
# harness models.yml id and the homepage probe do not move.
ALIAS=frank-large
# HIP reads a bare HOST, so this is BIND_ADDR — the same reason the file
# avoids a bare TEMP.
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770

# One full native window. llama.cpp divides --ctx-size across --parallel
# slots, so 262144/1 IS the model's trained window; the card validated 65,536
# and the fleet has been running this family at 262,144 since the gufo era.
CTX_SIZE=262144
PARALLEL=1

# Deck knobs. --ubatch-size has a measured optimum on this stack (512 peak on
# the sibling qwen35 model; larger ubatch costs GiBs of compute buffer for no
# speed), and --fit must stay off: --fit's auto placement does not know that
# 311 tensors of type 103 belong on the iGPU.
BATCH_SIZE=2048
UBATCH_SIZE=512
NGL=999
FIT=off
FLASH_ATTN=on
# Unquantized KV — the quality-first choice, affordable at one slot.
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16

# Speculative decode: the model ships a standalone MTP head (mtp-Q4_0.gguf,
# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 with n-min 0
# and p-min 0.0 is the card's measured optimum (+31% end to end, acceptance
# 0.926; its own n-max 8 measurement only reached 0.757 acceptance and LOST).
# The draft runs entirely on the iGPU, so cross-device traffic stays off the
# bandwidth-limited link.
SPEC_TYPE=draft-mtp
SPEC_DRAFT_NGL=99
SPEC_DRAFT_DEVICE=ROCm0
SPEC_DRAFT_N_MAX=4
SPEC_DRAFT_N_MIN=0
SPEC_DRAFT_P_MIN=0.0

# The card's thinking sampling profile, which is also what the harness sends
# in extraBody, so the endpoint behaves the same to a client that omits it.
TEMPERATURE=1.0
TOP_P=0.95
TOP_K=20
MIN_P=0.0
PRESENCE_PENALTY=0.0

NUM_THREADS=16
THREAD_BATCH=16