frank-large: swap to the uncensored Q6_0_ROCMFPX_AGENT payload
Frank's order: replace frank-large with kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF's Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf, following that card. The file is ftype 114 — 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159 GGML_TYPE_Q8_0_ROCMFPX (103) + 353 F32 over 851 tensors, qwen35/64 layers/full_attention_interval 4/4 KV heads/262,144 window — read out of its own header. The pinned ROCmFPX build already knows both types (ggml.h 102/103 plus convert/cpy/getrows/mmq/fattn device paths for 102), so no rebuild: the fork stays at 36e9acd and the payload directory is the only new host state. Card compliance, all already our flags: Q4_0 draft head (not Q8_0), --spec-draft-n-max 4 (the knee on both cards), --spec-draft-ngl 99, --jinja, --fit off, -fa on, -dio, one slot at full window, thinking with the card's own sampling triple (which the file also embeds as its general.sampling defaults). Kept from the deployed shape: the BF16 projector (this repo ships only a Q8_0 one; BF16 is the file verified 4/4 on spatial ground truth here) and the byte-identical MTP head, which the new script hardlinks instead of re-fetching 1.6 GiB. Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md
This commit is contained in:
1 parent
0ae923b879
commit
7d3dd33de0
5 files changed
+236
-91
No files matched your search
+48
-30
@@ -1,37 +1,47 @@
|
||||
# frank-large — ROCmFPX llama.cpp serving
|
||||
# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF (Q8_0_ROCMFPX_AGENT, ftype
|
||||
# 115) + its Q4_0 MTP draft head, on the Strix Halo iGPU (gfx1151), :7770.
|
||||
# kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
|
||||
# (Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, ftype 114) + its Q4_0 MTP draft
|
||||
# head, on the Strix Halo iGPU (gfx1151), :7770.
|
||||
#
|
||||
# 2026-10-02 (Frank): "the highest quality of that repo, one full context,
|
||||
# whatever runtime it requires". The highest-quality rung is Q8_0_ROCMFPX_AGENT
|
||||
# (26.28 GiB, 8.39 bpw: 187 tensors at plain Q8_0 + 311 at
|
||||
# GGML_TYPE_Q8_0_ROCMFPX); the AGENT routing is also the variant the card
|
||||
# tells you to take with MTP, because it keeps the output-head tensors at full
|
||||
# Q8_0 (187 vs 2), which is what the draft scores against. The runtime the
|
||||
# format REQUIRES is the ROCmFPX fork — type 103 exists nowhere else.
|
||||
# 2026-10-03 (Frank): swapped from the ALIGNED Q8_0_ROCMFPX_AGENT (ftype 115)
|
||||
# to this repo's Q6 AGENT rung — same card, same recipe, abliterated weights.
|
||||
# The card is blunt about what that buys: refusal counts go from 23/24 refuse
|
||||
# to 23/24 comply with the quality score byte-identical (6/8 both arms), and
|
||||
# "abliteration removes content-refusal. It does not add capability." It costs
|
||||
# one quant rung: 8.39 bpw → ~6.6 bpw, 26.28 GiB → 23.15 GiB.
|
||||
#
|
||||
# The shape: one session at the model's full native window (262,144), which is
|
||||
# the whole point — the Q8_0 head costs quality the fleet wanted back, and
|
||||
# the draft head pays for the bandwidth the bigger file spends.
|
||||
# What the file actually contains (read out of its own GGUF header, not from
|
||||
# the card): arch qwen35, 64 layers, full_attention_interval 4, 24 heads /
|
||||
# 4 KV heads / 256 head_dim, context_length 262144, and 851 tensors — 353 F32
|
||||
# + 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159 GGML_TYPE_Q8_0_ROCMFPX (103). The
|
||||
# AGENT routing is what puts the embeddings, the output head and every fourth
|
||||
# attention projection at the Q8 rung. Two custom tensor types, both fork-only
|
||||
# → the runtime is chosen by the file, exactly as it was for ftype 115.
|
||||
#
|
||||
# The shape is unchanged: one session at the model's full native window, MTP
|
||||
# drafting at n-max 4, unquantized KV.
|
||||
#
|
||||
# Budget (from the GGUF header, not from a boot log): attention KV lives only
|
||||
# in the 16 full-attention layers (4 KV heads x 256 head_dim), 64 KiB/token
|
||||
# with f16 K and V => 16.0 GiB at 262,144. The 48 Gated DeltaNet layers hold
|
||||
# ~144 MiB of fixed state per sequence. Weights 26.28 + draft head 1.57 +
|
||||
# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 45-47 GiB against the
|
||||
# ~144 MiB of fixed state per sequence. Weights 23.15 + draft head 1.57 +
|
||||
# mmproj 0.87 + KV 16.0 + state 0.14 + compute buffers ≈ 42-44 GiB against the
|
||||
# iGPU's ~114 GiB GTT aperture, so f16 KV (the unquantized choice) is affordable
|
||||
# here in a way it is not on frank-small's 24 GiB card.
|
||||
|
||||
# The qualified ROCmFPX fork build (scripts/build-rocmfpx.sh pins the commit).
|
||||
LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
|
||||
# One directory per HF repo, created by scripts/download-frank-large-kingjones.sh.
|
||||
MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
|
||||
# One directory per HF repo, created by
|
||||
# scripts/download-frank-large-kingjones-uncensored.sh.
|
||||
MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf
|
||||
# The draft head: Q4_0, NOT the Q8_0 head. The head's per-step cost is its
|
||||
# projection over a 248K vocabulary, so the cheaper head is faster end to end
|
||||
# at equal-or-better acceptance (card: +6.0% at n-max 1, +10.2% at n-max 2).
|
||||
DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
# Vision tower, separate from the LLM file. Empty string = text-only.
|
||||
MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf
|
||||
# The uncensored repo redistributes this file byte for byte (same Hub LFS oid);
|
||||
# the download script hardlinks the verified copy instead of re-fetching 1.6 GiB.
|
||||
DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
# Vision tower, separate from the LLM file. This repo ships a Q8_0 projector;
|
||||
# the fleet keeps the BF16 one, which is the file verified 4/4 on spatial
|
||||
# ground truth on this host (hardlinked here, same bytes). Empty = text-only.
|
||||
MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf
|
||||
|
||||
# Fleet endpoint: same port and alias as every frank-large before it, so the
|
||||
# harness models.yml id and the homepage probe do not move.
|
||||
@@ -42,15 +52,16 @@ BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# One full native window. llama.cpp divides --ctx-size across --parallel
|
||||
# slots, so 262144/1 IS the model's trained window; the card validated 65,536
|
||||
# and the fleet has been running this family at 262,144 since the gufo era.
|
||||
# slots, so 262144/1 IS the window the file itself declares
|
||||
# (qwen35.context_length = 262144); the card only validated 65,536 and the
|
||||
# fleet has been running this family at 262,144 since the gufo era.
|
||||
CTX_SIZE=262144
|
||||
PARALLEL=1
|
||||
|
||||
# Deck knobs. --ubatch-size has a measured optimum on this stack (512 peak on
|
||||
# the sibling qwen35 model; larger ubatch costs GiBs of compute buffer for no
|
||||
# speed), and --fit must stay off: --fit's auto placement does not know that
|
||||
# 311 tensors of type 103 belong on the iGPU.
|
||||
# 498 tensors of types 102 and 103 belong on the iGPU.
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=512
|
||||
NGL=999
|
||||
@@ -61,10 +72,11 @@ CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
|
||||
# Speculative decode: the model ships a standalone MTP head (mtp-Q4_0.gguf,
|
||||
# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 with n-min 0
|
||||
# and p-min 0.0 is the card's measured optimum (+31% end to end, acceptance
|
||||
# 0.926; its own n-max 8 measurement only reached 0.757 acceptance and LOST).
|
||||
# The draft runs entirely on the iGPU, so cross-device traffic stays off the
|
||||
# arch qwen35, nextn_predict_layers=1, a 65-block draft). n-max 4 is the knee
|
||||
# on BOTH cards: aligned 30.13 / 30.30 / 27.52 at n=3/4/5 (acceptance 0.926 at
|
||||
# the knee, and n-max 8 measured WORSE at 0.757), uncensored Q6 24.96 / 27.66
|
||||
# / 25.27. n-min 0 and p-min 0.0 are the card's own recipe. The draft runs
|
||||
# entirely on the iGPU, so cross-device traffic stays off the
|
||||
# bandwidth-limited link.
|
||||
SPEC_TYPE=draft-mtp
|
||||
SPEC_DRAFT_NGL=99
|
||||
@@ -73,8 +85,14 @@ SPEC_DRAFT_N_MAX=4
|
||||
SPEC_DRAFT_N_MIN=0
|
||||
SPEC_DRAFT_P_MIN=0.0
|
||||
|
||||
# The card's thinking sampling profile, which is also what the harness sends
|
||||
# in extraBody, so the endpoint behaves the same to a client that omits it.
|
||||
# The card's thinking sampling profile — the same triple the file embeds as
|
||||
# its own defaults (general.sampling.temp/top_p/top_k = 1.0 / 0.95 / 20) and
|
||||
# what the harness sends in extraBody, so the endpoint behaves the same to a
|
||||
# client that omits it. Card traps that do NOT bite here, but do not "fix"
|
||||
# them by changing this: reasoning_effort "none" THROWS (use
|
||||
# chat_template_kwargs enable_thinking:false), and a small max_tokens with
|
||||
# thinking on returns empty content — the serve script runs
|
||||
# --reasoning-budget -1, so the budget is never the limiter.
|
||||
TEMPERATURE=1.0
|
||||
TOP_P=0.95
|
||||
TOP_K=20
|
||||
|
||||
Reference in new issue
Block a user