frank-default-large: re-enable --flash-attn on via QSA F32-mask patch
fa on aborted graph build: build_attn_qsa called ggml_fill on the F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the MLA sparse path in llm_graph_context::build_attn (F32 base mask for the top-k surgery, F16 cast only at the attention call, gated on cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch (sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with it and validated live at CTX_SIZE=32768 (health 200, real completion).
This commit is contained in:
1 parent
101b1f8be2
commit
7cb6eeb67a
5 files changed
+70
-21
No files matched your search
@@ -10,12 +10,16 @@
|
||||
#
|
||||
# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
|
||||
# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
|
||||
# KNOWN CONSTRAINTS on the charlie12345/ROCmFPX build-qwen38 runtime.
|
||||
# 1. FA must stay OFF. --flash-attn on aborts graph build: the QSA
|
||||
# indexer path calls ggml_fill on the F16-cast kq mask (ggml_fill
|
||||
# asserts GGML_TYPE_F32 at ggml.c:5371). The v2 main loads and serves
|
||||
# fine with --flash-attn off (verified live: model loaded, health 200,
|
||||
# real completion).
|
||||
# 1. Flash attention is ON. fa on used to abort graph build: the QSA
|
||||
# indexer path called ggml_fill on the F16-cast kq mask (ggml_fill
|
||||
# asserts GGML_TYPE_F32 at ggml.c:5371). Fixed in the fork source by
|
||||
# the QSA F32-mask patch (scripts/patches/qwen4exp-qsa-f32-mask.patch,
|
||||
# sha256 f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f)
|
||||
# and the rebuilt binary at LLAMA_BIN already carries the fix. A clean
|
||||
# rebuild must re-apply: git apply scripts/patches/qwen4exp-qsa-f32-mask.patch
|
||||
# inside /home/dank/GitPackages/ROCmFPX before make. Verified live:
|
||||
# --flash-attn on + v2 main at CTX_SIZE=32768 loads, health 200, real
|
||||
# completion.
|
||||
# 2. The MTP draft head does NOT load on this binary. The qwen4exp loader
|
||||
# requires blk.0.hc_attn_norm.weight per layer; the agentionai FAST
|
||||
# head carries only blk.48 (verified 3x: EasiiX Q8_0 + FAST against
|
||||
@@ -54,7 +58,7 @@ ALIAS=frank-default-large
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# Serving: 131072-token context on ONE slot (parallel 1), batch 4096 /
|
||||
# Serving: 32768-token context on ONE slot (parallel 1), batch 4096 /
|
||||
# ubatch 4096, 16+16 threads, f16 KV, 16 context checkpoints, 8 GiB
|
||||
# prompt cache. --fit off: auto-fit segfaults on this model's
|
||||
# hyper-connection/indexer config; -ngl 999 puts every layer on the
|
||||
@@ -81,7 +85,7 @@ SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state
|
||||
SPLIT_MODE=none
|
||||
FIT=off
|
||||
NGL=999
|
||||
FLASH_ATTN=off
|
||||
FLASH_ATTN=on
|
||||
|
||||
# Sampling (card instruct table, non-thinking).
|
||||
TEMPERATURE=0.7
|
||||
|
||||
Reference in new issue
Block a user