frank-default-large: verify v2 main on build-qwen38 — flash-attn off required, MTP draft incompatible (ngram-mod fallback)
- Main model (v2 joined PLE layout, sha256 1e4160ca…) loads and serves on build-qwen38 only with --flash-attn off: the QSA indexer path calls ggml_fill on the F16-cast kq mask and aborts graph build (GGML_ASSERT F32 at ggml.c:5371). Verified live: model loaded, health 200, real completion. - MTP draft head (agentionai FAST, 2.28 GiB) does not load on this binary: qwen4exp loader requires full per-layer blk.<il>.hc_attn_norm.weight, head carries only blk.48. Third confirmation (EasiiX Q8_0 + agentionai FAST vs kingjones STRIX_LEAN previously, now own-matched pair). Spec decoding runs ngram-mod only; head retained for a future compatible build. - FLASH_ATTN=off and SPEC_TYPE=ngram-mod wired through env + serve script; runbook and AGENTS.md updated with the verified constraints.
This commit is contained in:
Frank
@@ -10,25 +10,20 @@
|
||||
#
|
||||
# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
|
||||
# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
|
||||
# at layer 1) quantized Q4_0_ROCMFP4_FAST imatrix-calibrated, 87.06 GiB,
|
||||
# 4.23 bpw. The n-gram PLE table is split per head inside the single GGUF
|
||||
# (fully VRAM-resident — no ple sidecar, no --ngram-on-disk). Vision tower
|
||||
# (f16, lossless) ships as mmproj/. The model's own MTP head (2.28 GiB,
|
||||
# matched ROCmFP4-FAST quant, ~66% draft acceptance) drives spec decoding.
|
||||
# KNOWN CONSTRAINTS on the charlie12345/ROCmFPX build-qwen38 runtime.
|
||||
# 1. FA must stay OFF. --flash-attn on aborts graph build: the QSA
|
||||
# indexer path calls ggml_fill on the F16-cast kq mask (ggml_fill
|
||||
# asserts GGML_TYPE_F32 at ggml.c:5371). The v2 main loads and serves
|
||||
# fine with --flash-attn off (verified live: model loaded, health 200,
|
||||
# real completion).
|
||||
# 2. The MTP draft head does NOT load on this binary. The qwen4exp loader
|
||||
# requires blk.0.hc_attn_norm.weight per layer; the agentionai FAST
|
||||
# head carries only blk.48 (verified 3x: EasiiX Q8_0 + FAST against
|
||||
# kingjones STRIX_LEAN, FAST against this v2 main). Speculative
|
||||
# decoding therefore runs ngram-mod only. The 2.28 GiB head stays on
|
||||
# disk in case a compatible binary/head pair appears.
|
||||
#
|
||||
# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0,
|
||||
# presence 1.5 — non-thinking instruct) mixed with the validated
|
||||
# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096,
|
||||
# t 16 / tb 16 (threads-batch 16 measured best for prefill), f16 KV, fa on,
|
||||
# cont-batching, prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx
|
||||
# checkpoints, --fit off (auto-fit segfaults on qwen4exp
|
||||
# hyper-connection/indexer config). MTP depth 3 (n-max 3 measured best on
|
||||
# this head; adaptive p-split measured worse, left off). The Qwen-Sharp
|
||||
# chat template is kept on purpose (chat-template.jinja): the service keeps
|
||||
# emitting reasoning in DeepSeek format for its clients.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
|
||||
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
||||
# incident). llama.cpp reads --host/--port/--temp as CLI flags, never as
|
||||
# env vars.
|
||||
|
||||
@@ -37,11 +32,14 @@ LLAMA_BIN=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server
|
||||
|
||||
# Model package from
|
||||
# https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF
|
||||
# Layout: single main GGUF (per-head PLE table, fully VRAM-resident) +
|
||||
# mmproj/ vision tower. The MTP draft head lives in
|
||||
# /home/dank/models/mtp-Qwen3.8-Flash-Next-FAST (already on the host).
|
||||
MODEL_DIR=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST
|
||||
MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf
|
||||
# Layout: v2/ joined-layout GGUF (PLE n-gram table as a single
|
||||
# per_layer_token_embd.weight tensor — the layout the charlie12345/ROCmFPX
|
||||
# qwen4exp loader requires; the root -ple16 file splits the table per head
|
||||
# and only loads in the LaurentZuijdwijk Vulkan fork). mmproj/ vision tower.
|
||||
# The MTP draft head lives in /home/dank/models/mtp-Qwen3.8-Flash-Next-FAST
|
||||
# (already on the host).
|
||||
MODEL_DIR=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2
|
||||
MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf
|
||||
MMPROJ_PATH=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf
|
||||
|
||||
# Qwen-Sharp chat template (tracked in the repo, staged on the host). The
|
||||
@@ -72,9 +70,8 @@ CACHE_TYPE_V=f16
|
||||
CTX_CHECKPOINTS=16
|
||||
CACHE_RAM=8192
|
||||
SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state
|
||||
SPLIT_MODE=none
|
||||
FIT=off
|
||||
NGL=999
|
||||
|
||||
FLASH_ATTN=off
|
||||
|
||||
# Sampling (card instruct table, non-thinking).
|
||||
TEMPERATURE=0.7
|
||||
@@ -95,14 +92,16 @@ HSA_OVERRIDE_GFX_VERSION=11.5.1
|
||||
|
||||
VERBOSITY=4
|
||||
|
||||
# Speculative decoding: the model's own MTP head (agentionai
|
||||
# Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF, 2.28 GiB, ROCmFP4-FAST quant
|
||||
# matched to the target) at depth 3 stacked on ngram-mod. n-max 3 measured
|
||||
# best (32.4 t/s, 0.612 acceptance); adaptive p-split measured worse, so it
|
||||
# stays off (p-min stays at the 0.75 default). Set SPEC_TYPE=none to disable.
|
||||
# Speculative decoding: ngram-mod only. The MTP draft head is retained
|
||||
# (agentionai Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF, 2.28 GiB) but does
|
||||
# NOT load on build-qwen38 (loader requires blk.0.hc_attn_norm.weight the
|
||||
# head lacks — see KNOWN CONSTRAINTS). If a compatible binary appears, set
|
||||
# SPEC_TYPE=draft-mtp,ngram-mod (n-max 3 measured best at 32.4 t/s / 0.612
|
||||
# acceptance; adaptive p-split measured worse, stays off). SPEC_TYPE=none
|
||||
# disables speculative decoding entirely.
|
||||
DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST
|
||||
DRAFT_MODEL_FILE=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf
|
||||
SPEC_TYPE=draft-mtp,ngram-mod
|
||||
SPEC_TYPE=ngram-mod
|
||||
SPEC_DRAFT_N_MAX=3
|
||||
SPEC_DRAFT_N_MIN=0
|
||||
SPEC_DRAFT_P_MIN=0.75
|
||||
|
||||
@@ -12,18 +12,18 @@
|
||||
# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
|
||||
# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
|
||||
# at layer 1) quantized Q4_0_ROCMFP4_FAST, imatrix-calibrated, 87.06 GiB.
|
||||
# The n-gram PLE table is split per head inside the single GGUF — fully
|
||||
# VRAM-resident, no sidecar, no --ngram-on-disk. The model's own MTP
|
||||
# head (2.28 GiB, matched ROCmFP4-FAST quant) drives speculative decoding
|
||||
# at depth 3 (~66% draft acceptance).
|
||||
# The n-gram PLE table is a single per_layer_token_embd.weight tensor —
|
||||
# fully VRAM-resident, no sidecar, no --ngram-on-disk.
|
||||
#
|
||||
# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0,
|
||||
# presence 1.5 — non-thinking instruct) mixed with the validated
|
||||
# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096,
|
||||
# t 16 / tb 16, f16 KV, fa on, cont-batching, prompt cache + 8 GiB
|
||||
# cache-ram + idle-slot cache, 16 ctx checkpoints, --fit off (auto-fit
|
||||
# segfaults on qwen4exp hyper-connection config). MTP depth 3 (n-max 3
|
||||
# measured best; adaptive p-split measured worse, left off).
|
||||
# t 16 / tb 16, f16 KV, fa OFF (see KNOWN CONSTRAINTS in the env file:
|
||||
# flash-attn on aborts graph build on the QSA indexer path), cont-batching,
|
||||
# prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx checkpoints,
|
||||
# --fit off (auto-fit segfaults on qwen4exp hyper-connection config).
|
||||
# Speculative decoding is ngram-mod only; the MTP head does not load on
|
||||
# build-qwen38 (loader requires blk.0.hc_attn_norm.weight it lacks).
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -37,9 +37,9 @@ fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST}"
|
||||
: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf}"
|
||||
: "${MMPROJ_PATH:=$MODEL_DIR/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
|
||||
: "${MODEL_DIR:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2}"
|
||||
: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf}"
|
||||
: "${MMPROJ_PATH:=$MODEL_DIR/../mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
|
||||
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
|
||||
: "${CHAT_TEMPLATE_KWARGS:={}}"
|
||||
: "${ALIAS:=frank-default-large}"
|
||||
@@ -57,17 +57,15 @@ fi
|
||||
: "${CACHE_RAM:=8192}"
|
||||
: "${SLOT_SAVE_PATH:=$SCRIPT_DIR/slot-state}"
|
||||
: "${SPLIT_MODE:=none}"
|
||||
: "${FIT:=off}"
|
||||
: "${NGL:=999}"
|
||||
: "${FLASH_ATTN:=off}"
|
||||
: "${TEMPERATURE:=0.7}"
|
||||
: "${TOP_K:=20}"
|
||||
: "${TOP_P:=0.8}"
|
||||
: "${MIN_P:=0.0}"
|
||||
: "${PRESENCE_PENALTY:=1.5}"
|
||||
: "${REASONING:=on}"
|
||||
: "${REASONING_FORMAT:=deepseek}"
|
||||
: "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}"
|
||||
: "${SPEC_TYPE:=draft-mtp,ngram-mod}"
|
||||
: "${SPEC_TYPE:=ngram-mod}"
|
||||
: "${DRAFT_MODEL_DIR:=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST}"
|
||||
: "${DRAFT_MODEL_FILE:=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf}"
|
||||
: "${SPEC_DRAFT_N_MAX:=3}"
|
||||
@@ -154,7 +152,7 @@ exec /usr/bin/env \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--fit "$FIT" \
|
||||
--flash-attn on \
|
||||
--flash-attn "$FLASH_ATTN" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
--threads "$NUM_THREADS" \
|
||||
|
||||
Reference in New Issue
Block a user