# frank-large — ROCmFPX llama.cpp serving
# kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
# (Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, ftype 114) + its Q4_0 MTP draft
# head, on the Strix Halo iGPU (gfx1151), :7770.
#
# 2026-10-03 (Frank): swapped from the ALIGNED Q8_0_ROCMFPX_AGENT (ftype 115)
# to this repo's Q6 AGENT rung — same card, same recipe, abliterated weights.
# The card is blunt about what that buys: refusal counts go from 23/24 refuse
# to 23/24 comply with the quality score byte-identical (6/8 both arms), and
# "abliteration removes content-refusal. It does not add capability." It costs
# one quant rung: 8.39 bpw → ~6.6 bpw, 26.28 GiB → 23.15 GiB.
#
# What the file actually contains (read out of its own GGUF header, not from
# the card): arch qwen35, 64 layers, full_attention_interval 4, 24 heads /
# 4 KV heads / 256 head_dim, context_length 262144, and 851 tensors — 353 F32
# + 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159 GGML_TYPE_Q8_0_ROCMFPX (103). The
# AGENT routing is what puts the embeddings, the output head and every fourth
# attention projection at the Q8 rung. Two custom tensor types, both fork-only
# → the runtime is chosen by the file, exactly as it was for ftype 115.
#
# The shape is unchanged: one session at the model's full native window, MTP
# drafting at n-max 4, unquantized KV.
#
# Budget (from the GGUF header, not from a boot log): attention KV lives only
# in the 16 full-attention layers (4 KV heads x 256 head_dim), 64 KiB/token
# with f16 K and V => 16.0 GiB at 262,144. The 48 Gated DeltaNet layers hold
# ~144 MiB of fixed state per sequence. Weights 23.15 + draft head 1.57 +
# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 42-44 GiB against the
# iGPU's ~114 GiB GTT aperture, so f16 KV (the unquantized choice) is affordable
# here in a way it is not on frank-small's 24 GiB card.
LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
# One directory per HF repo, created by
# scripts/download-frank-large-kingjones-uncensored.sh.
MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf
# The draft head: Q4_0, NOT the Q8_0 head. The head's per-step cost is its
# projection over a 248K vocabulary, so the cheaper head is faster end to end
# at equal-or-better acceptance (card: +6.0% at n-max 1, +10.2% at n-max 2).
# The uncensored repo redistributes this file byte for byte (same Hub LFS oid);
# the download script hardlinks the verified copy instead of re-fetching 1.6 GiB.
DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
# Vision tower, separate from the LLM file. This repo ships a Q8_0 projector;
# the fleet keeps the BF16 one, which is the file verified 4/4 on spatial
# ground truth on this host (hardlinked here, same bytes). Empty = text-only.
MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf

# Fleet endpoint: same port and alias as every frank-large before it, so the
# harness models.yml id and the homepage probe do not move.
ALIAS=frank-large
# HIP reads a bare HOST, so this is BIND_ADDR — the same reason the file
# avoids a bare TEMP.
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770

# One full native window. llama.cpp divides --ctx-size across --parallel
# slots, so 262144/1 IS the window the file itself declares
# (qwen35.context_length = 262144); the card only validated 65,536 and the
# fleet has been running this family at 262,144 since the gufo era.
CTX_SIZE=262144
PARALLEL=1

# Deck knobs. --ubatch-size has a measured optimum on this stack (512 peak on
# the sibling qwen35 model; larger ubatch costs GiBs of compute buffer for no
# speed), and --fit must stay off: --fit's auto placement does not know that
# 498 tensors of types 102 and 103 belong on the iGPU.
BATCH_SIZE=2048
UBATCH_SIZE=512
NGL=999
FIT=off
FLASH_ATTN=on
# Unquantized KV — the quality-first choice, affordable at one slot.
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16

# Speculative decode: the model ships a standalone MTP head (mtp-Q4_0.gguf,
# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 is the knee
# on BOTH cards: aligned 30.13 / 30.30 / 27.52 at n=3/4/5 (acceptance 0.926 at
# the knee, and n-max 8 measured WORSE at 0.757), uncensored Q6 24.96 / 27.66
# / 25.27. n-min 0 and p-min 0.0 are the card's own recipe. The draft runs
# entirely on the iGPU, so cross-device traffic stays off the
# bandwidth-limited link.
SPEC_TYPE=draft-mtp
SPEC_DRAFT_NGL=99
SPEC_DRAFT_DEVICE=ROCm0
SPEC_DRAFT_N_MAX=4
SPEC_DRAFT_N_MIN=0
SPEC_DRAFT_P_MIN=0.0

# The card's thinking sampling profile — the same triple the file embeds as
# its own defaults (general.sampling.temp/top_p/top_k = 1.0 / 0.95 / 20) and
# what the harness sends in extraBody, so the endpoint behaves the same to a
# client that omits it. Card traps that do NOT bite here, but do not "fix"
# them by changing this: reasoning_effort "none" THROWS (use
# chat_template_kwargs enable_thinking:false), and a small max_tokens with
# thinking on returns empty content — the serve script runs
# --reasoning-budget -1, so the budget is never the limiter.
TEMPERATURE=1.0
TOP_P=0.95
TOP_K=20
MIN_P=0.0
PRESENCE_PENALTY=0.0

NUM_THREADS=16
THREAD_BATCH=16