frank-large: swap to the uncensored Q6_0_ROCMFPX_AGENT payload

Frank's order: replace frank-large with
kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF's
Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf, following that card.

The file is ftype 114 — 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159
GGML_TYPE_Q8_0_ROCMFPX (103) + 353 F32 over 851 tensors,
qwen35/64 layers/full_attention_interval 4/4 KV heads/262,144 window —
read out of its own header. The pinned ROCmFPX build already knows both
types (ggml.h 102/103 plus convert/cpy/getrows/mmq/fattn device paths
for 102), so no rebuild: the fork stays at 36e9acd and the payload
directory is the only new host state.

Card compliance, all already our flags: Q4_0 draft head (not Q8_0),
--spec-draft-n-max 4 (the knee on both cards), --spec-draft-ngl 99,
--jinja, --fit off, -fa on, -dio, one slot at full window, thinking with
the card's own sampling triple (which the file also embeds as its
general.sampling defaults). Kept from the deployed shape: the BF16
projector (this repo ships only a Q8_0 one; BF16 is the file verified
4/4 on spatial ground truth here) and the byte-identical MTP head, which
the new script hardlinks instead of re-fetching 1.6 GiB.

Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md
This commit is contained in:
Frank committed 2026-10-03 02:04:22 -07:00
1 parent 0ae923b879
commit 7d3dd33de0
5 files changed
+236 -91

No files matched your search

+48 -30
View File
@@ -1,37 +1,47 @@
# frank-large — ROCmFPX llama.cpp serving
# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF (Q8_0_ROCMFPX_AGENT, ftype
# 115) + its Q4_0 MTP draft head, on the Strix Halo iGPU (gfx1151), :7770.
# kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
# (Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, ftype 114) + its Q4_0 MTP draft
# head, on the Strix Halo iGPU (gfx1151), :7770.
#
# 2026-10-02 (Frank): "the highest quality of that repo, one full context,
# whatever runtime it requires". The highest-quality rung is Q8_0_ROCMFPX_AGENT
# (26.28 GiB, 8.39 bpw: 187 tensors at plain Q8_0 + 311 at
# GGML_TYPE_Q8_0_ROCMFPX); the AGENT routing is also the variant the card
# tells you to take with MTP, because it keeps the output-head tensors at full
# Q8_0 (187 vs 2), which is what the draft scores against. The runtime the
# format REQUIRES is the ROCmFPX fork — type 103 exists nowhere else.
# 2026-10-03 (Frank): swapped from the ALIGNED Q8_0_ROCMFPX_AGENT (ftype 115)
# to this repo's Q6 AGENT rung — same card, same recipe, abliterated weights.
# The card is blunt about what that buys: refusal counts go from 23/24 refuse
# to 23/24 comply with the quality score byte-identical (6/8 both arms), and
# "abliteration removes content-refusal. It does not add capability." It costs
# one quant rung: 8.39 bpw → ~6.6 bpw, 26.28 GiB → 23.15 GiB.
#
# The shape: one session at the model's full native window (262,144), which is
# the whole point — the Q8_0 head costs quality the fleet wanted back, and
# the draft head pays for the bandwidth the bigger file spends.
# What the file actually contains (read out of its own GGUF header, not from
# the card): arch qwen35, 64 layers, full_attention_interval 4, 24 heads /
# 4 KV heads / 256 head_dim, context_length 262144, and 851 tensors — 353 F32
# + 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159 GGML_TYPE_Q8_0_ROCMFPX (103). The
# AGENT routing is what puts the embeddings, the output head and every fourth
# attention projection at the Q8 rung. Two custom tensor types, both fork-only
# → the runtime is chosen by the file, exactly as it was for ftype 115.
#
# The shape is unchanged: one session at the model's full native window, MTP
# drafting at n-max 4, unquantized KV.
#
# Budget (from the GGUF header, not from a boot log): attention KV lives only
# in the 16 full-attention layers (4 KV heads x 256 head_dim), 64 KiB/token
# with f16 K and V => 16.0 GiB at 262,144. The 48 Gated DeltaNet layers hold
# ~144 MiB of fixed state per sequence. Weights 26.28 + draft head 1.57 +
# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 45-47 GiB against the
# ~144 MiB of fixed state per sequence. Weights 23.15 + draft head 1.57 +
# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 42-44 GiB against the
# iGPU's ~114 GiB GTT aperture, so f16 KV (the unquantized choice) is affordable
# here in a way it is not on frank-small's 24 GiB card.
# The qualified ROCmFPX fork build (scripts/build-rocmfpx.sh pins the commit).
LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
# One directory per HF repo, created by scripts/download-frank-large-kingjones.sh.
MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
# One directory per HF repo, created by
# scripts/download-frank-large-kingjones-uncensored.sh.
MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf
# The draft head: Q4_0, NOT the Q8_0 head. The head's per-step cost is its
# projection over a 248K vocabulary, so the cheaper head is faster end to end
# at equal-or-better acceptance (card: +6.0% at n-max 1, +10.2% at n-max 2).
DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
# Vision tower, separate from the LLM file. Empty string = text-only.
MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf
# The uncensored repo redistributes this file byte for byte (same Hub LFS oid);
# the download script hardlinks the verified copy instead of re-fetching 1.6 GiB.
DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
# Vision tower, separate from the LLM file. This repo ships a Q8_0 projector;
# the fleet keeps the BF16 one, which is the file verified 4/4 on spatial
# ground truth on this host (hardlinked here, same bytes). Empty = text-only.
MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf
# Fleet endpoint: same port and alias as every frank-large before it, so the
# harness models.yml id and the homepage probe do not move.
@@ -42,15 +52,16 @@ BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# One full native window. llama.cpp divides --ctx-size across --parallel
# slots, so 262144/1 IS the model's trained window; the card validated 65,536
# and the fleet has been running this family at 262,144 since the gufo era.
# slots, so 262144/1 IS the window the file itself declares
# (qwen35.context_length = 262144); the card only validated 65,536 and the
# fleet has been running this family at 262,144 since the gufo era.
CTX_SIZE=262144
PARALLEL=1
# Deck knobs. --ubatch-size has a measured optimum on this stack (512 peak on
# the sibling qwen35 model; larger ubatch costs GiBs of compute buffer for no
# speed), and --fit must stay off: --fit's auto placement does not know that
# 311 tensors of type 103 belong on the iGPU.
# 498 tensors of types 102 and 103 belong on the iGPU.
BATCH_SIZE=2048
UBATCH_SIZE=512
NGL=999
@@ -61,10 +72,11 @@ CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
# Speculative decode: the model ships a standalone MTP head (mtp-Q4_0.gguf,
# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 with n-min 0
# and p-min 0.0 is the card's measured optimum (+31% end to end, acceptance
# 0.926; its own n-max 8 measurement only reached 0.757 acceptance and LOST).
# The draft runs entirely on the iGPU, so cross-device traffic stays off the
# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 is the knee
# on BOTH cards: aligned 30.13 / 30.30 / 27.52 at n=3/4/5 (acceptance 0.926 at
# the knee, and n-max 8 measured WORSE at 0.757), uncensored Q6 24.96 / 27.66
# / 25.27. n-min 0 and p-min 0.0 are the card's own recipe. The draft runs
# entirely on the iGPU, so cross-device traffic stays off the
# bandwidth-limited link.
SPEC_TYPE=draft-mtp
SPEC_DRAFT_NGL=99
@@ -73,8 +85,14 @@ SPEC_DRAFT_N_MAX=4
SPEC_DRAFT_N_MIN=0
SPEC_DRAFT_P_MIN=0.0
# The card's thinking sampling profile, which is also what the harness sends
# in extraBody, so the endpoint behaves the same to a client that omits it.
# The card's thinking sampling profile — the same triple the file embeds as
# its own defaults (general.sampling.temp/top_p/top_k = 1.0 / 0.95 / 20) and
# what the harness sends in extraBody, so the endpoint behaves the same to a
# client that omits it. Card traps that do NOT bite here, but do not "fix"
# them by changing this: reasoning_effort "none" THROWS (use
# chat_template_kwargs enable_thinking:false), and a small max_tokens with
# thinking on returns empty content — the serve script runs
# --reasoning-budget -1, so the budget is never the limiter.
TEMPERATURE=1.0
TOP_P=0.95
TOP_K=20