Frank
125d04967f
The repo's highest-quality rung is ftype 115: 311 tensors of GGML_TYPE_Q8_0_ROCMFPX, a type only the ROCmFPX fork reads, so the runtime moves from gufo to scripts/build-rocmfpx.sh's pinned fork build (36e9acd) — the first build on this host measured to load this file without the 2026-08-25 set_rows segfault. One session at the native 262,144 window, f16 KV (64 KiB/token, 16 GiB), the repo's Q4_0 MTP head drafting 4 tokens, plus the bundled BF16 projector so the endpoint has vision again. The serve script now honors caller environment overrides over the env file (a plain source let the file win, which made single-run overrides no-ops) and defaults apply only when a value is unset, so an explicit LLAMA_BIN='' is the error it reads as.
86 lines
4.0 KiB
Bash
86 lines
4.0 KiB
Bash
# frank-large — ROCmFPX llama.cpp serving
|
|
# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF (Q8_0_ROCMFPX_AGENT, ftype
|
|
# 115) + its Q4_0 MTP draft head, on the Strix Halo iGPU (gfx1151), :7770.
|
|
#
|
|
# 2026-10-02 (Frank): "the highest quality of that repo, one full context,
|
|
# whatever runtime it requires". The highest-quality rung is Q8_0_ROCMFPX_AGENT
|
|
# (26.28 GiB, 8.39 bpw: 187 tensors at plain Q8_0 + 311 at
|
|
# GGML_TYPE_Q8_0_ROCMFPX); the AGENT routing is also the variant the card
|
|
# tells you to take with MTP, because it keeps the output-head tensors at full
|
|
# Q8_0 (187 vs 2), which is what the draft scores against. The runtime the
|
|
# format REQUIRES is the ROCmFPX fork — type 103 exists nowhere else.
|
|
#
|
|
# The shape: one session at the model's full native window (262,144), which is
|
|
# the whole point — the Q8_0 head costs quality the fleet wanted back, and
|
|
# the draft head pays for the bandwidth the bigger file spends.
|
|
#
|
|
# Budget (from the GGUF header, not from a boot log): attention KV lives only
|
|
# in the 16 full-attention layers (4 KV heads x 256 head_dim), 64 KiB/token
|
|
# with f16 K and V => 16.0 GiB at 262,144. The 48 Gated DeltaNet layers hold
|
|
# ~144 MiB of fixed state per sequence. Weights 26.28 + draft head 1.57 +
|
|
# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 45-47 GiB against the
|
|
# iGPU's ~114 GiB GTT aperture, so f16 KV (the unquantized choice) is affordable
|
|
# here in a way it is not on frank-small's 24 GiB card.
|
|
|
|
# The qualified ROCmFPX fork build (scripts/build-rocmfpx.sh pins the commit).
|
|
LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
|
|
# One directory per HF repo, created by scripts/download-frank-large-kingjones.sh.
|
|
MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
|
|
# The draft head: Q4_0, NOT the Q8_0 head. The head's per-step cost is its
|
|
# projection over a 248K vocabulary, so the cheaper head is faster end to end
|
|
# at equal-or-better acceptance (card: +6.0% at n-max 1, +10.2% at n-max 2).
|
|
DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
|
|
# Vision tower, separate from the LLM file. Empty string = text-only.
|
|
MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-large before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-large
|
|
# HIP reads a bare HOST, so this is BIND_ADDR — the same reason the file
|
|
# avoids a bare TEMP.
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7770
|
|
|
|
# One full native window. llama.cpp divides --ctx-size across --parallel
|
|
# slots, so 262144/1 IS the model's trained window; the card validated 65,536
|
|
# and the fleet has been running this family at 262,144 since the gufo era.
|
|
CTX_SIZE=262144
|
|
PARALLEL=1
|
|
|
|
# Deck knobs. --ubatch-size has a measured optimum on this stack (512 peak on
|
|
# the sibling qwen35 model; larger ubatch costs GiBs of compute buffer for no
|
|
# speed), and --fit must stay off: --fit's auto placement does not know that
|
|
# 311 tensors of type 103 belong on the iGPU.
|
|
BATCH_SIZE=2048
|
|
UBATCH_SIZE=512
|
|
NGL=999
|
|
FIT=off
|
|
FLASH_ATTN=on
|
|
# Unquantized KV — the quality-first choice, affordable at one slot.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
|
|
# Speculative decode: the model ships a standalone MTP head (mtp-Q4_0.gguf,
|
|
# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 with n-min 0
|
|
# and p-min 0.0 is the card's measured optimum (+31% end to end, acceptance
|
|
# 0.926; its own n-max 8 measurement only reached 0.757 acceptance and LOST).
|
|
# The draft runs entirely on the iGPU, so cross-device traffic stays off the
|
|
# bandwidth-limited link.
|
|
SPEC_TYPE=draft-mtp
|
|
SPEC_DRAFT_NGL=99
|
|
SPEC_DRAFT_DEVICE=ROCm0
|
|
SPEC_DRAFT_N_MAX=4
|
|
SPEC_DRAFT_N_MIN=0
|
|
SPEC_DRAFT_P_MIN=0.0
|
|
|
|
# The card's thinking sampling profile, which is also what the harness sends
|
|
# in extraBody, so the endpoint behaves the same to a client that omits it.
|
|
TEMPERATURE=1.0
|
|
TOP_P=0.95
|
|
TOP_K=20
|
|
MIN_P=0.0
|
|
PRESENCE_PENALTY=0.0
|
|
|
|
NUM_THREADS=16
|
|
THREAD_BATCH=16
|