dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-large: swap to the uncensored Q6_0_ROCMFPX_AGENT payload

Browse Source 
Frank's order: replace frank-large with
kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF's
Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf, following that card.

The file is ftype 114 — 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159
GGML_TYPE_Q8_0_ROCMFPX (103) + 353 F32 over 851 tensors,
qwen35/64 layers/full_attention_interval 4/4 KV heads/262,144 window —
read out of its own header. The pinned ROCmFPX build already knows both
types (ggml.h 102/103 plus convert/cpy/getrows/mmq/fattn device paths
for 102), so no rebuild: the fork stays at 36e9acd and the payload
directory is the only new host state.

Card compliance, all already our flags: Q4_0 draft head (not Q8_0),
--spec-draft-n-max 4 (the knee on both cards), --spec-draft-ngl 99,
--jinja, --fit off, -fa on, -dio, one slot at full window, thinking with
the card's own sampling triple (which the file also embeds as its
general.sampling defaults). Kept from the deployed shape: the BF16
projector (this repo ships only a Q8_0 one; BF16 is the file verified
4/4 on spatial ground truth here) and the byte-identical MTP head, which
the new script hardlinks instead of re-fetching 1.6 GiB.

Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md
This commit is contained in:
Frank
2026-10-03 02:04:22 -07:00
parent 0ae923b879
commit 7d3dd33de0
5 changed files with 236 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_large_unit.bats
+18 -17
View File
@@ -1,12 +1,13 @@
#!/usr/bin/env bats
# Seam: frank-large serves kingjones777's Qwen3.8-27B Q8_0_ROCMFPX_AGENT (a
# ftype-115 ROCmFPX file) with the repo's MTP draft head on the Strix Halo
# iGPU, via the ROCmFPX fork of llama.cpp — one session at the native
# 262,144 window.
# Seam: frank-large serves kingjones777's UNCENSORED Qwen3.8-27B
# Q6_0_ROCMFPX_AGENT (a ftype-114 ROCmFPX file) with the repo's MTP draft head
# on the Strix Halo iGPU, via the ROCmFPX fork of llama.cpp — one session at
# the native 262,144 window.
#
# This seam exists because the format is the constraint: GGML_TYPE_Q8_0_ROCMFPX
# (tensor type 103) is understood by the ROCmFPX fork and by nothing else on
# this host, so the runtime, the payload and the deployer's gate have to agree
# This seam exists because the format is the constraint: GGML_TYPE_Q6_0_ROCMFPX
# (tensor type 102) and GGML_TYPE_Q8_0_ROCMFPX (103) are understood by the
# ROCmFPX fork and by nothing else on this host, so the runtime, the payload
# and the deployer's gate have to agree
# on paths that no other endpoint uses. On 2026-10-01 a serve-script gate and
# its deployer twin disagreed during a cutover and the endpoint crash-looped;
# both sides are pinned here.
@@ -23,9 +24,9 @@ setup() {
DEPLOYER="$REPO_ROOT/scripts/deploy-remote.sh"
BUILD="$REPO_ROOT/scripts/build-rocmfpx.sh"
RUNTIME_STUB="/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server"
MODEL_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf"
DRAFT_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf"
MMPROJ_STUB="/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf"
MODEL_STUB="/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf"
DRAFT_STUB="/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf"
MMPROJ_STUB="/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf"
TEST_TMP="$(mktemp -d)"
SRC="$TEST_TMP/src"
@@ -76,7 +77,7 @@ STUB
"DRAFT_FILE=$TEST_TMP/draft.gguf" "MMPROJ_FILE=$TEST_TMP/mmproj.gguf" "$SERVE"
[ "$status" -ne 0 ]
[[ "$output" == *"is not readable"* ]]
[[ "$output" == *"download-frank-large-kingjones.sh"* ]]
[[ "$output" == *"download-frank-large-kingjones-uncensored.sh"* ]]
}
@test "the serve script refuses to start without the MTP draft head" {
@@ -139,10 +140,10 @@ STUB
@test "the shipped argv is one full native window on one session" {
grep -qE '^CTX_SIZE=262144$' "$ENV_FILE"
grep -qE '^PARALLEL=1$' "$ENV_FILE"
# 262144 IS the model's trained window (general.context_length in the
# GGUF), so 1 slot gets the whole thing — no truncation, no yarn scaling.
# 262144 IS the model's trained window (qwen35.context_length in the GGUF),
# so 1 slot gets the whole thing — no truncation, no yarn scaling.
#
# Budget, from the GGUF header: 26.28 GiB of weights + 1.57 GiB draft head
# Budget, from the GGUF header: 23.15 GiB of weights + 1.57 GiB draft head
# + 0.87 GiB projector + 16 full-attention layers x 4 kv heads x 256 dim x
# 2 (K+V) x 2 B = 64 KiB/token of f16 KV (~16 GiB at 262,144) + ~0.14 GiB
# of DeltaNet state + a couple GiB of compute buffers, against the iGPU's
@@ -150,7 +151,7 @@ STUB
# log, is what tells you it no longer fits.
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
kv_gib=$((ctx * 65536 / 1073741824))
total_gib=$((27 + 2 + 1 + kv_gib + 3))
total_gib=$((24 + 2 + 1 + kv_gib + 3))
[ "$total_gib" -le 114 ]
}
@@ -230,14 +231,14 @@ STUB
@test "the deployer points a missing runtime at the fork build script" {
grep -qE 'build it: bash scripts/build-rocmfpx\.sh' "$DEPLOYER"
grep -qE 'fetch it: bash scripts/download-frank-large-kingjones\.sh' "$DEPLOYER"
grep -qE 'fetch it: bash scripts/download-frank-large-kingjones-uncensored\.sh' "$DEPLOYER"
}
@test "the stray-process sweep cannot take frank-small down with frank-large" {
# frank-small serves a different GGUF over llama-server on the SAME host.
# A sweep keyed on the binary name (`pkill -f llama-server`) would kill
# the 3090 endpoint too, so it is keyed on frank-large's own payload file.
grep -q "pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'" "$DEPLOYER"
grep -q "pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'" "$DEPLOYER"
! grep -qE "pkill -f 'llama-server'" "$DEPLOYER"
}