diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 81e21e4..9aa279c 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -1,49 +1,51 @@ -# Tunables for frank-default-large (agentionai/Qwen3.8-Flash-Next-ROCmFP4- -# FAST-imatrix-GGUF on the charlie12345/ROCmFPX llama.cpp runtime, -# build-qwen38, Dankclaw, Strix Halo gfx1151 unified memory). +# Tunables for frank-default-large (kingjones777/Qwen3.8-Flash-Next-ROCmFP4- +# STRIX-GGUF on the kingjones30/ROCmFPX llama.cpp runtime, Dankclaw, +# Strix Halo gfx1151, unified memory). # -# Runtime: charlie12345/ROCmFPX llama.cpp at -# /home/dank/GitPackages/ROCmFPX/build-qwen38 (HIP, ROCm 7.2.4, gfx1151, -# PR #27742 qwen4exp merged). The ROCmFP4 quant types are hand-ported from -# ciru-ai/ROCmFPX; upstream llama.cpp cannot load this GGUF. Replaced the -# jcbtc CIRU v1.1 (ciru-ai runtime) and kingjones STRIX_LEAN deployments. +# Runtime: kingjones30/ROCmFPX (fork of charlie12345/ROCmFPX with the +# qwen4exp-on-rocmfpx-d3ca537 patch applied — the model card's pinned +# runtime: ROCmFP4 tensor types AND the qwen4exp architecture in one +# tree; stock llama.cpp and charlie's fork alone cannot load this GGUF). +# Built per the card recipe: +# cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release +# at /home/dank/GitPackages/kingjones-ROCmFPX/build. +# Replaced the agentionai ROCmFP4-FAST imatrix deployment (charlie's +# build-qwen38 runtime) on 2026-08-30. # # Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used, # 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE -# 1. Flash attention is ON. fa on used to abort graph build: the QSA -# indexer path called ggml_fill on the F16-cast kq mask (ggml_fill -# asserts GGML_TYPE_F32 at ggml.c:5371). Fixed in the fork source by -# the QSA F32-mask patch (scripts/patches/qwen4exp-qsa-f32-mask.patch, -# sha256 f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f) -# and the rebuilt binary at LLAMA_BIN already carries the fix. A clean -# rebuild must re-apply: git apply scripts/patches/qwen4exp-qsa-f32-mask.patch -# inside /home/dank/GitPackages/ROCmFPX before make. Verified live: -# --flash-attn on + v2 main at CTX_SIZE=32768 loads, health 200, real -# completion. -# 2. The MTP draft head does NOT load on this binary. The qwen4exp loader -# requires blk.0.hc_attn_norm.weight per layer; the agentionai FAST -# head carries only blk.48 (verified 3x: EasiiX Q8_0 + FAST against -# kingjones STRIX_LEAN, FAST against this v2 main). Speculative -# decoding therefore runs ngram-mod only. The 2.28 GiB head stays on -# disk in case a compatible binary/head pair appears. +# at layer 1), Q4_0_ROCMFP4_STRIX (file_type 105), 113.47 GiB single +# file. STRIX is the top quality tier: PLE table at Q8_0 (50.66 GiB), +# token embeddings + lm head at Q6_K. The PLE table NEVER enters GPU +# memory — it streams from the file through the OS page cache. Two hard +# card constraints: +# 1. mmap MUST stay enabled (default) — --no-mmap forces the 50.66 GiB +# table into anonymous memory and the cgroup killer takes the +# process with nothing in the server log (only dmesg). +# 2. Do NOT pin the PLE tensor to CPU with -ot — measured gen fell +# ~23 -> 13.4 tok/s; the kernel streams it better. +# NOTE: the card's own build_attn_qsa still calls ggml_fill on the +# F16-cast kq mask (the path that aborted our charlie build with +# --flash-attn on before scripts/patches/qwen4exp-qsa-f32-mask.patch). +# If a clean kingjones-tree build aborts graph build with +# GGML_ASSERT(a-type == GGML_TYPE_F32), port that fix (F32 base mask + +# cast at the attention call) into this tree and rebuild. # # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP # incident). llama.cpp reads --host/--port/--temp as CLI flags, never as # env vars. -# ROCmFPX (build-qwen38) runtime binary. -LLAMA_BIN=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server +# kingjones30/ROCmFPX runtime binary (card recipe build). +LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server # Model package from -# https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF -# Layout: v2/ joined-layout GGUF (PLE n-gram table as a single -# per_layer_token_embd.weight tensor — the layout the charlie12345/ROCmFPX -# qwen4exp loader requires; the root -ple16 file splits the table per head -# and only loads in the LaurentZuijdwijk Vulkan fork). mmproj/ vision tower. -# The MTP draft head lives in /home/dank/models/mtp-Qwen3.8-Flash-Next-FAST -# (already on the host). -MODEL_DIR=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2 -MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf +# https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF +# Layout: single unsharded GGUF, text-only (the repo ships no mmproj; +# the vision tower below is borrowed from the agentionai FAST package — +# same base model, f16 — kept only if it loads clean, drop +# MMPROJ_PATH if it does not). +MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX +MODEL_FILE=Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf MMPROJ_PATH=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf # Qwen-Sharp chat template (tracked in the repo, staged on the host). The @@ -62,11 +64,14 @@ LISTEN_PORT=7770 # ubatch 512, 16+16 threads, f16 KV, 16 context checkpoints, 8 GiB # prompt cache. --fit off: auto-fit segfaults on this model's # hyper-connection/indexer config; -ngl 999 puts every layer on the -# SVM-backed GPU pool. +# SVM-backed GPU pool. ubatch 512 matches the card's re-measurement +# (-ub 1024/2048/4096 showed no effect; -t 32 vs 16 is noise). # -# CTX_SIZE=401408 / PARALLEL=2: untested experiment. At 32768×1 the -# context buffer was ~13 GiB and total ~100 GiB in the 119.3 GiB HIP -# heap. Doubling slots × 12× context could OOM — verify load. +# CTX_SIZE=401408 / PARALLEL=2: carried over from the FAST deployment +# where it loaded clean (~74 s, health 200). The card measures the full +# 262144 window at ~71.7 GiB GTT at load on the same 128 GB board; QSA +# caps KV growth so the extra window is cheap. KV footprint scales +# with the 12 attention layers only. Verify load on first STRIX boot. CTX_SIZE=401408 PARALLEL=2 @@ -104,7 +109,11 @@ HSA_OVERRIDE_GFX_VERSION=11.5.1 VERBOSITY=4 -# Speculative decoding: disabled. +# Speculative decoding: disabled. The card's Usage block ships no spec +# flags. The Flash-Next MTP heads do not load on this loader (requires +# per-layer blk.N.hc_attn_norm; every head carries only blk.48 — +# verified 3x on build-qwen38). DRAFT_MODEL_* point at the retired +# agentionai FAST head and are unused while SPEC_TYPE=none. DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST DRAFT_MODEL_FILE=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf SPEC_TYPE=none diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 36a36c1..16a0b33 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -1,29 +1,31 @@ #!/usr/bin/env bash -# Serve agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF as -# frank-default-large via the charlie12345/ROCmFPX llama.cpp runtime -# (build-qwen38, PR #27742 qwen4exp merged) on Dankclaw (Strix Halo -# gfx1151, unified memory). +# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF as +# frank-default-large via the kingjones30/ROCmFPX llama.cpp runtime on +# Dankclaw (Strix Halo gfx1151, unified memory). # -# Runtime: charlie12345/ROCmFPX llama.cpp built at -# /home/dank/GitPackages/ROCmFPX/build-qwen38 (HIP, gfx1151, ROCm 7.2.4). -# The ROCmFP4 quant types (Q4_0_ROCMFP4_FAST) are hand-ported from -# ciru-ai/ROCmFPX; upstream llama.cpp cannot load this GGUF. +# Runtime: kingjones30/ROCmFPX (fork of charlie12345/ROCmFPX with the +# model card's qwen4exp-on-rocmfpx patch) built at +# /home/dank/GitPackages/kingjones-ROCmFPX/build (HIP, gfx1151, +# ROCm 7.2.4, card recipe). The ROCmFP4 quant types plus the qwen4exp +# architecture must be in one tree; stock llama.cpp cannot load this +# GGUF. # # Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used, # 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE -# at layer 1) quantized Q4_0_ROCMFP4_FAST, imatrix-calibrated, 87.06 GiB. -# The n-gram PLE table is a single per_layer_token_embd.weight tensor — -# fully VRAM-resident, no sidecar, no --ngram-on-disk. +# at layer 1) quantized Q4_0_ROCMFP4_STRIX, 113.47 GiB single file. The +# 50.66 GiB PLE n-gram table stays OUT of GPU memory and streams via +# mmap from the OS page cache — never add --no-mmap or -ot for it. # # Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0, -# presence 1.5 — non-thinking instruct) mixed with the validated -# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096, -# t 16 / tb 16, f16 KV, fa OFF (see KNOWN CONSTRAINTS in the env file: -# flash-attn on aborts graph build on the QSA indexer path), cont-batching, -# prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx checkpoints, -# --fit off (auto-fit segfaults on qwen4exp hyper-connection config). -# Speculative decoding is ngram-mod only; the MTP head does not load on -# build-qwen38 (loader requires blk.0.hc_attn_norm.weight it lacks). +# presence 1.5 — non-thinking instruct) mixed with the validated serving +# profile: ctx 401408 x 2 slots, b 4096 / ub 512, t 16 / tb 16, f16 KV, +# fa on (if the graph build aborts on the QSA F16 mask, see the note in +# the env file: port scripts/patches/qwen4exp-qsa-f32-mask.patch into +# this tree and rebuild), cont-batching, prompt cache + 8 GiB cache-ram +# + idle-slot cache, 16 ctx checkpoints, --fit off (auto-fit segfaults +# on qwen4exp hyper-connection config). Speculative decoding off: the +# Flash-Next MTP heads do not load on this loader; the card ships no +# spec flags. set -euo pipefail SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" @@ -36,10 +38,10 @@ if [[ -f "$ENV_FILE" ]]; then fi # --- defaults (used when the env file does not set them) ------------------- -: "${LLAMA_BIN:=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server}" -: "${MODEL_DIR:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2}" -: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf}" -: "${MMPROJ_PATH:=$MODEL_DIR/../mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}" +: "${LLAMA_BIN:=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}" +: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX}" +: "${MODEL_FILE:=Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf}" +: "${MMPROJ_PATH:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}" : "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}" : "${CHAT_TEMPLATE_KWARGS:={}}" : "${ALIAS:=frank-default-large}" @@ -122,8 +124,8 @@ if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then fi fi -# The agentionai FAST package includes the model's own vision tower (f16, -# lossless). Pass --mmproj only when MMPROJ_PATH points at a readable file. +# The STRIX repo ships no mmproj; this is the agentionai FAST package's +# f16 vision tower for the same base model. Passed only when readable. MMPROJ_ARGS=() if [[ -n "${MMPROJ_PATH:-}" && -r "${MMPROJ_PATH:-}" ]]; then MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")