Frank
7d3dd33de0
Frank's order: replace frank-large with kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF's Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf, following that card. The file is ftype 114 — 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159 GGML_TYPE_Q8_0_ROCMFPX (103) + 353 F32 over 851 tensors, qwen35/64 layers/full_attention_interval 4/4 KV heads/262,144 window — read out of its own header. The pinned ROCmFPX build already knows both types (ggml.h 102/103 plus convert/cpy/getrows/mmq/fattn device paths for 102), so no rebuild: the fork stays at 36e9acd and the payload directory is the only new host state. Card compliance, all already our flags: Q4_0 draft head (not Q8_0), --spec-draft-n-max 4 (the knee on both cards), --spec-draft-ngl 99, --jinja, --fit off, -fa on, -dio, one slot at full window, thinking with the card's own sampling triple (which the file also embeds as its general.sampling defaults). Kept from the deployed shape: the BF16 projector (this repo ships only a Q8_0 one; BF16 is the file verified 4/4 on spatial ground truth here) and the byte-identical MTP head, which the new script hardlinks instead of re-fetching 1.6 GiB. Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md
104 lines
5.2 KiB
Bash
104 lines
5.2 KiB
Bash
# frank-large — ROCmFPX llama.cpp serving
|
|
# kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
|
|
# (Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, ftype 114) + its Q4_0 MTP draft
|
|
# head, on the Strix Halo iGPU (gfx1151), :7770.
|
|
#
|
|
# 2026-10-03 (Frank): swapped from the ALIGNED Q8_0_ROCMFPX_AGENT (ftype 115)
|
|
# to this repo's Q6 AGENT rung — same card, same recipe, abliterated weights.
|
|
# The card is blunt about what that buys: refusal counts go from 23/24 refuse
|
|
# to 23/24 comply with the quality score byte-identical (6/8 both arms), and
|
|
# "abliteration removes content-refusal. It does not add capability." It costs
|
|
# one quant rung: 8.39 bpw → ~6.6 bpw, 26.28 GiB → 23.15 GiB.
|
|
#
|
|
# What the file actually contains (read out of its own GGUF header, not from
|
|
# the card): arch qwen35, 64 layers, full_attention_interval 4, 24 heads /
|
|
# 4 KV heads / 256 head_dim, context_length 262144, and 851 tensors — 353 F32
|
|
# + 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159 GGML_TYPE_Q8_0_ROCMFPX (103). The
|
|
# AGENT routing is what puts the embeddings, the output head and every fourth
|
|
# attention projection at the Q8 rung. Two custom tensor types, both fork-only
|
|
# → the runtime is chosen by the file, exactly as it was for ftype 115.
|
|
#
|
|
# The shape is unchanged: one session at the model's full native window, MTP
|
|
# drafting at n-max 4, unquantized KV.
|
|
#
|
|
# Budget (from the GGUF header, not from a boot log): attention KV lives only
|
|
# in the 16 full-attention layers (4 KV heads x 256 head_dim), 64 KiB/token
|
|
# with f16 K and V => 16.0 GiB at 262,144. The 48 Gated DeltaNet layers hold
|
|
# ~144 MiB of fixed state per sequence. Weights 23.15 + draft head 1.57 +
|
|
# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 42-44 GiB against the
|
|
# iGPU's ~114 GiB GTT aperture, so f16 KV (the unquantized choice) is affordable
|
|
# here in a way it is not on frank-small's 24 GiB card.
|
|
LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
|
|
# One directory per HF repo, created by
|
|
# scripts/download-frank-large-kingjones-uncensored.sh.
|
|
MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf
|
|
# The draft head: Q4_0, NOT the Q8_0 head. The head's per-step cost is its
|
|
# projection over a 248K vocabulary, so the cheaper head is faster end to end
|
|
# at equal-or-better acceptance (card: +6.0% at n-max 1, +10.2% at n-max 2).
|
|
# The uncensored repo redistributes this file byte for byte (same Hub LFS oid);
|
|
# the download script hardlinks the verified copy instead of re-fetching 1.6 GiB.
|
|
DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
|
|
# Vision tower, separate from the LLM file. This repo ships a Q8_0 projector;
|
|
# the fleet keeps the BF16 one, which is the file verified 4/4 on spatial
|
|
# ground truth on this host (hardlinked here, same bytes). Empty = text-only.
|
|
MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-large before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-large
|
|
# HIP reads a bare HOST, so this is BIND_ADDR — the same reason the file
|
|
# avoids a bare TEMP.
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7770
|
|
|
|
# One full native window. llama.cpp divides --ctx-size across --parallel
|
|
# slots, so 262144/1 IS the window the file itself declares
|
|
# (qwen35.context_length = 262144); the card only validated 65,536 and the
|
|
# fleet has been running this family at 262,144 since the gufo era.
|
|
CTX_SIZE=262144
|
|
PARALLEL=1
|
|
|
|
# Deck knobs. --ubatch-size has a measured optimum on this stack (512 peak on
|
|
# the sibling qwen35 model; larger ubatch costs GiBs of compute buffer for no
|
|
# speed), and --fit must stay off: --fit's auto placement does not know that
|
|
# 498 tensors of types 102 and 103 belong on the iGPU.
|
|
BATCH_SIZE=2048
|
|
UBATCH_SIZE=512
|
|
NGL=999
|
|
FIT=off
|
|
FLASH_ATTN=on
|
|
# Unquantized KV — the quality-first choice, affordable at one slot.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
|
|
# Speculative decode: the model ships a standalone MTP head (mtp-Q4_0.gguf,
|
|
# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 is the knee
|
|
# on BOTH cards: aligned 30.13 / 30.30 / 27.52 at n=3/4/5 (acceptance 0.926 at
|
|
# the knee, and n-max 8 measured WORSE at 0.757), uncensored Q6 24.96 / 27.66
|
|
# / 25.27. n-min 0 and p-min 0.0 are the card's own recipe. The draft runs
|
|
# entirely on the iGPU, so cross-device traffic stays off the
|
|
# bandwidth-limited link.
|
|
SPEC_TYPE=draft-mtp
|
|
SPEC_DRAFT_NGL=99
|
|
SPEC_DRAFT_DEVICE=ROCm0
|
|
SPEC_DRAFT_N_MAX=4
|
|
SPEC_DRAFT_N_MIN=0
|
|
SPEC_DRAFT_P_MIN=0.0
|
|
|
|
# The card's thinking sampling profile — the same triple the file embeds as
|
|
# its own defaults (general.sampling.temp/top_p/top_k = 1.0 / 0.95 / 20) and
|
|
# what the harness sends in extraBody, so the endpoint behaves the same to a
|
|
# client that omits it. Card traps that do NOT bite here, but do not "fix"
|
|
# them by changing this: reasoning_effort "none" THROWS (use
|
|
# chat_template_kwargs enable_thinking:false), and a small max_tokens with
|
|
# thinking on returns empty content — the serve script runs
|
|
# --reasoning-budget -1, so the budget is never the limiter.
|
|
TEMPERATURE=1.0
|
|
TOP_P=0.95
|
|
TOP_K=20
|
|
MIN_P=0.0
|
|
PRESENCE_PENALTY=0.0
|
|
|
|
NUM_THREADS=16
|
|
THREAD_BATCH=16
|