frank-default-large: swap to kingjones STRIX model on card-pinned kingjones30/ROCmFPX runtime
- runtime: charlie build-qwen38 -> kingjones30/ROCmFPX (card recipe build, LLAMA_BUILD_WEBUI=OFF since npm/HF-bucket provisioning fails; all 35 of our serving flags verified present in the tree) - model: agentionai FAST imatrix -> kingjones777 STRIX single GGUF (on-host copy sha256 == HF LFS oid c6770d74...49fa verified) - args unchanged from our validated profile (401408x2, b4096/ub512, t16/tb16, f16 KV, fa on, fit off); mmap stays default per card (PLE streams via page cache; --no-mmap = silent cgroup OOM)
This commit is contained in:
1 parent
f69efcdde6
commit
3066d810ee
2 files changed
+76
-65
No files matched your search
@@ -1,29 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF as
|
||||
# frank-default-large via the charlie12345/ROCmFPX llama.cpp runtime
|
||||
# (build-qwen38, PR #27742 qwen4exp merged) on Dankclaw (Strix Halo
|
||||
# gfx1151, unified memory).
|
||||
# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF as
|
||||
# frank-default-large via the kingjones30/ROCmFPX llama.cpp runtime on
|
||||
# Dankclaw (Strix Halo gfx1151, unified memory).
|
||||
#
|
||||
# Runtime: charlie12345/ROCmFPX llama.cpp built at
|
||||
# /home/dank/GitPackages/ROCmFPX/build-qwen38 (HIP, gfx1151, ROCm 7.2.4).
|
||||
# The ROCmFP4 quant types (Q4_0_ROCMFP4_FAST) are hand-ported from
|
||||
# ciru-ai/ROCmFPX; upstream llama.cpp cannot load this GGUF.
|
||||
# Runtime: kingjones30/ROCmFPX (fork of charlie12345/ROCmFPX with the
|
||||
# model card's qwen4exp-on-rocmfpx patch) built at
|
||||
# /home/dank/GitPackages/kingjones-ROCmFPX/build (HIP, gfx1151,
|
||||
# ROCm 7.2.4, card recipe). The ROCmFP4 quant types plus the qwen4exp
|
||||
# architecture must be in one tree; stock llama.cpp cannot load this
|
||||
# GGUF.
|
||||
#
|
||||
# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
|
||||
# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
|
||||
# at layer 1) quantized Q4_0_ROCMFP4_FAST, imatrix-calibrated, 87.06 GiB.
|
||||
# The n-gram PLE table is a single per_layer_token_embd.weight tensor —
|
||||
# fully VRAM-resident, no sidecar, no --ngram-on-disk.
|
||||
# at layer 1) quantized Q4_0_ROCMFP4_STRIX, 113.47 GiB single file. The
|
||||
# 50.66 GiB PLE n-gram table stays OUT of GPU memory and streams via
|
||||
# mmap from the OS page cache — never add --no-mmap or -ot for it.
|
||||
#
|
||||
# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0,
|
||||
# presence 1.5 — non-thinking instruct) mixed with the validated
|
||||
# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096,
|
||||
# t 16 / tb 16, f16 KV, fa OFF (see KNOWN CONSTRAINTS in the env file:
|
||||
# flash-attn on aborts graph build on the QSA indexer path), cont-batching,
|
||||
# prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx checkpoints,
|
||||
# --fit off (auto-fit segfaults on qwen4exp hyper-connection config).
|
||||
# Speculative decoding is ngram-mod only; the MTP head does not load on
|
||||
# build-qwen38 (loader requires blk.0.hc_attn_norm.weight it lacks).
|
||||
# presence 1.5 — non-thinking instruct) mixed with the validated serving
|
||||
# profile: ctx 401408 x 2 slots, b 4096 / ub 512, t 16 / tb 16, f16 KV,
|
||||
# fa on (if the graph build aborts on the QSA F16 mask, see the note in
|
||||
# the env file: port scripts/patches/qwen4exp-qsa-f32-mask.patch into
|
||||
# this tree and rebuild), cont-batching, prompt cache + 8 GiB cache-ram
|
||||
# + idle-slot cache, 16 ctx checkpoints, --fit off (auto-fit segfaults
|
||||
# on qwen4exp hyper-connection config). Speculative decoding off: the
|
||||
# Flash-Next MTP heads do not load on this loader; the card ships no
|
||||
# spec flags.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -36,10 +38,10 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2}"
|
||||
: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf}"
|
||||
: "${MMPROJ_PATH:=$MODEL_DIR/../mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX}"
|
||||
: "${MODEL_FILE:=Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf}"
|
||||
: "${MMPROJ_PATH:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
|
||||
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
|
||||
: "${CHAT_TEMPLATE_KWARGS:={}}"
|
||||
: "${ALIAS:=frank-default-large}"
|
||||
@@ -122,8 +124,8 @@ if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
# The agentionai FAST package includes the model's own vision tower (f16,
|
||||
# lossless). Pass --mmproj only when MMPROJ_PATH points at a readable file.
|
||||
# The STRIX repo ships no mmproj; this is the agentionai FAST package's
|
||||
# f16 vision tower for the same base model. Passed only when readable.
|
||||
MMPROJ_ARGS=()
|
||||
if [[ -n "${MMPROJ_PATH:-}" && -r "${MMPROJ_PATH:-}" ]]; then
|
||||
MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
|
||||
|
||||
Reference in new issue
Block a user