frank-large: swap to the uncensored Q6_0_ROCMFPX_AGENT payload
Frank's order: replace frank-large with kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF's Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf, following that card. The file is ftype 114 — 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159 GGML_TYPE_Q8_0_ROCMFPX (103) + 353 F32 over 851 tensors, qwen35/64 layers/full_attention_interval 4/4 KV heads/262,144 window — read out of its own header. The pinned ROCmFPX build already knows both types (ggml.h 102/103 plus convert/cpy/getrows/mmq/fattn device paths for 102), so no rebuild: the fork stays at 36e9acd and the payload directory is the only new host state. Card compliance, all already our flags: Q4_0 draft head (not Q8_0), --spec-draft-n-max 4 (the knee on both cards), --spec-draft-ngl 99, --jinja, --fit off, -fa on, -dio, one slot at full window, thinking with the card's own sampling triple (which the file also embeds as its general.sampling defaults). Kept from the deployed shape: the BF16 projector (this repo ships only a Q8_0 one; BF16 is the file verified 4/4 on spatial ground truth here) and the byte-identical MTP head, which the new script hardlinks instead of re-fetching 1.6 GiB. Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md
This commit is contained in:
Frank committed
1 parent
0ae923b879
commit
7d3dd33de0
5 files changed
+236
-91
No files matched your search
@@ -1,26 +1,32 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF as frank-large on
|
||||
# Dankclaw's Strix Halo iGPU (gfx1151) with the ROCmFPX fork of llama.cpp.
|
||||
# Serve kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF as
|
||||
# frank-large on Dankclaw's Strix Halo iGPU (gfx1151) with the ROCmFPX fork of
|
||||
# llama.cpp.
|
||||
#
|
||||
# 2026-10-02 (Frank): swapped from gufo + Q4_K_XL to the highest-quality file
|
||||
# in the repo, one session at the native 262,144 window.
|
||||
# 2026-10-03 (Frank): swapped from the ALIGNED Q8_0_ROCMFPX_AGENT to this
|
||||
# repo's Q6_0_ROCMFPX_AGENT — abliterated weights, one quant rung cheaper,
|
||||
# everything else (window, KV, MTP depth, sampler) unchanged.
|
||||
#
|
||||
# Why this runtime and not the previous one: the payload is ftype 115 — 311
|
||||
# tensors of GGML_TYPE_Q8_0_ROCMFPX (103), a tensor type that exists only in
|
||||
# the ROCmFPX fork. Stock llama.cpp, SGLang and gufo all refuse the file, and
|
||||
# gufo additionally gates every "Qwen3.8" artifact on a template-hash
|
||||
# allowlist. The qualified build is scripts/build-rocmfpx.sh's pin: it loads
|
||||
# ftype 115, survives warmup (the 2026-08-25 builds of the older fork line
|
||||
# segfaulted in ggml_compute_forward_set_rows on this exact file), and drafts
|
||||
# with the model's own MTP head.
|
||||
# Why this runtime and not another: the payload is ftype 114 — 339 tensors of
|
||||
# GGML_TYPE_Q6_0_ROCMFPX (102) and 159 of GGML_TYPE_Q8_0_ROCMFPX (103), tensor
|
||||
# types that exist only in the ROCmFPX fork. Stock llama.cpp, SGLang and gufo
|
||||
# all refuse the file, and gufo additionally gates every "Qwen3.8" artifact on
|
||||
# a template-hash allowlist. The qualified build is scripts/build-rocmfpx.sh's
|
||||
# pin: it defines both types (ggml.h 102/103), it carries the device paths for
|
||||
# type 102 (convert/cpy/getrows/mmq/fattn in ggml-cuda), it survives warmup —
|
||||
# the 2026-08-25 builds of the older fork line segfaulted in
|
||||
# ggml_compute_forward_set_rows on this family — and it drafts with the
|
||||
# model's own MTP head.
|
||||
#
|
||||
# Why this file: Q8_0_ROCMFPX_AGENT is the top of the repo's quality ladder
|
||||
# (26.28 GiB, 8.39 bpw) and the AGENT routing keeps 187 tensors — the
|
||||
# output-head set the draft scores against — at plain Q8_0 instead of the 2
|
||||
# the non-AGENT build keeps, which is the variant the card names for MTP
|
||||
# serving. Drafting uses the repo's Q4_0 MTP head: the head's cost is its
|
||||
# projection over the 248K vocabulary, so the 4-bit head is faster end to end
|
||||
# at equal-or-better acceptance than its Q8_0 sibling.
|
||||
# Why this file: Q6_0_ROCMFPX_AGENT (23.15 GiB, ftype 114) is the card's
|
||||
# protected-head rung — "Take Q6 AGENT (114) if you want protected heads": the
|
||||
# AGENT routing keeps the embeddings, the output head and every fourth
|
||||
# attention projection at the Q8 rung (159 tensors), which is the variant the
|
||||
# card names for MTP serving, and the refusal study was scored on this exact
|
||||
# file (23/24 comply where the aligned Q8 AGENT refuses 23/24, quality score
|
||||
# identical at 6/8). Drafting uses the repo's Q4_0 MTP head: the head's cost is
|
||||
# its projection over the 248K vocabulary, so the 4-bit head is faster end to
|
||||
# end at equal-or-better acceptance than its Q8_0 sibling.
|
||||
#
|
||||
# The client contract lives here, not in frank-large.env:
|
||||
# * --jinja — without it llama.cpp silently IGNORES the harness's
|
||||
@@ -64,9 +70,9 @@ done
|
||||
unset _var _kv
|
||||
|
||||
: "${LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
|
||||
: "${MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
|
||||
: "${DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
|
||||
: "${MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
|
||||
: "${MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf}"
|
||||
: "${DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
|
||||
: "${MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
|
||||
: "${ALIAS=frank-large}"
|
||||
: "${BIND_ADDR=0.0.0.0}"
|
||||
: "${LISTEN_PORT=7770}"
|
||||
@@ -106,14 +112,15 @@ unset _var _kv
|
||||
# makes llama-server hang rather than fail clean.
|
||||
[[ -r "$MODEL_FILE" ]] || {
|
||||
echo "frank-large: model file '$MODEL_FILE' is not readable;" >&2
|
||||
echo " fetch it with scripts/download-frank-large-kingjones.sh" >&2
|
||||
echo " fetch it with scripts/download-frank-large-kingjones-uncensored.sh" >&2
|
||||
exit 1
|
||||
}
|
||||
# The MTP head is the performance feature: without it the Q8_0 weights decode
|
||||
# at memory-bandwidth speed and the endpoint loses the ~1.4x the head buys.
|
||||
# The MTP head is the performance feature: without it these weights decode at
|
||||
# memory-bandwidth speed (measured on the aligned sibling: 17.0 t/s with the
|
||||
# head against 7.6 without) and the endpoint loses the ~2.2x the head buys.
|
||||
[[ -r "$DRAFT_FILE" ]] || {
|
||||
echo "frank-large: MTP draft head '$DRAFT_FILE' is not readable;" >&2
|
||||
echo " fetch it with scripts/download-frank-large-kingjones.sh" >&2
|
||||
echo " fetch it with scripts/download-frank-large-kingjones-uncensored.sh" >&2
|
||||
exit 1
|
||||
}
|
||||
# The GGUF carries no vision tower; the projector file is the whole tower, so
|
||||
|
||||
Reference in new issue
Block a user