frank-default-large: re-enable --flash-attn on via QSA F32-mask patch
fa on aborted graph build: build_attn_qsa called ggml_fill on the F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the MLA sparse path in llm_graph_context::build_attn (F32 base mask for the top-k surgery, F16 cast only at the attention call, gated on cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch (sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with it and validated live at CTX_SIZE=32768 (health 200, real completion).
This commit is contained in:
Frank
@@ -298,9 +298,10 @@ slot (author quick-start; the fully GPU-resident 87 GiB model + f16 KV +
|
||||
compute buffer OOMs the SVM pool beyond that — verified live: cudaMalloc
|
||||
fails at 131072/262144), b 4096 / ub 4096, t 16 / tb 16, f16 KV cache,
|
||||
`--fit off` (auto-fit segfaults on qwen4exp hyper-connection config),
|
||||
`--flash-attn off` (required: fa on aborts graph build on the QSA indexer
|
||||
path — F32 `ggml_fill` on the F16 kq mask, ggml.c:5371), and reasoning on (Sharp
|
||||
template, budget -1, `--reasoning-format deepseek`; chat kwargs keep
|
||||
`--flash-attn on` (restored 2026-08-29 via the QSA F32-mask patch,
|
||||
`scripts/patches/qwen4exp-qsa-f32-mask.patch` — re-apply inside
|
||||
`/home/dank/GitPackages/ROCmFPX` before any clean rebuild), and reasoning on
|
||||
(Sharp template, budget -1, `--reasoning-format deepseek`; chat kwargs keep
|
||||
`preserve_thinking` + `preserve_reasoning`). Env
|
||||
`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`.
|
||||
|
||||
@@ -309,7 +310,10 @@ model (v2 joined PLE layout), replacing the kingjones STRIX and jcbtc CIRU v1.1
|
||||
deployments. The own-matched MTP draft head does NOT load on build-qwen38 —
|
||||
verified against the v2 main itself (`missing tensor blk.0.hc_attn_norm.weight`),
|
||||
third confirmation (EasiiX Q8_0 + agentionai FAST vs STRIX_LEAN earlier). The
|
||||
head stays on disk for a future compatible binary.
|
||||
head stays on disk for a future compatible binary. Same day, `--flash-attn on`
|
||||
was restored: the QSA F32-mask patch (vendored
|
||||
`scripts/patches/qwen4exp-qsa-f32-mask.patch`) fixes the F16-kq_mask
|
||||
`ggml_fill` abort; the rebuilt fork binary carries it, verified live at 32K.
|
||||
Runbook:
|
||||
[serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md).
|
||||
|
||||
|
||||
@@ -10,12 +10,16 @@
|
||||
#
|
||||
# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
|
||||
# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
|
||||
# KNOWN CONSTRAINTS on the charlie12345/ROCmFPX build-qwen38 runtime.
|
||||
# 1. FA must stay OFF. --flash-attn on aborts graph build: the QSA
|
||||
# indexer path calls ggml_fill on the F16-cast kq mask (ggml_fill
|
||||
# asserts GGML_TYPE_F32 at ggml.c:5371). The v2 main loads and serves
|
||||
# fine with --flash-attn off (verified live: model loaded, health 200,
|
||||
# real completion).
|
||||
# 1. Flash attention is ON. fa on used to abort graph build: the QSA
|
||||
# indexer path called ggml_fill on the F16-cast kq mask (ggml_fill
|
||||
# asserts GGML_TYPE_F32 at ggml.c:5371). Fixed in the fork source by
|
||||
# the QSA F32-mask patch (scripts/patches/qwen4exp-qsa-f32-mask.patch,
|
||||
# sha256 f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f)
|
||||
# and the rebuilt binary at LLAMA_BIN already carries the fix. A clean
|
||||
# rebuild must re-apply: git apply scripts/patches/qwen4exp-qsa-f32-mask.patch
|
||||
# inside /home/dank/GitPackages/ROCmFPX before make. Verified live:
|
||||
# --flash-attn on + v2 main at CTX_SIZE=32768 loads, health 200, real
|
||||
# completion.
|
||||
# 2. The MTP draft head does NOT load on this binary. The qwen4exp loader
|
||||
# requires blk.0.hc_attn_norm.weight per layer; the agentionai FAST
|
||||
# head carries only blk.48 (verified 3x: EasiiX Q8_0 + FAST against
|
||||
@@ -54,7 +58,7 @@ ALIAS=frank-default-large
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# Serving: 131072-token context on ONE slot (parallel 1), batch 4096 /
|
||||
# Serving: 32768-token context on ONE slot (parallel 1), batch 4096 /
|
||||
# ubatch 4096, 16+16 threads, f16 KV, 16 context checkpoints, 8 GiB
|
||||
# prompt cache. --fit off: auto-fit segfaults on this model's
|
||||
# hyper-connection/indexer config; -ngl 999 puts every layer on the
|
||||
@@ -81,7 +85,7 @@ SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state
|
||||
SPLIT_MODE=none
|
||||
FIT=off
|
||||
NGL=999
|
||||
FLASH_ATTN=off
|
||||
FLASH_ATTN=on
|
||||
|
||||
# Sampling (card instruct table, non-thinking).
|
||||
TEMPERATURE=0.7
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
--- a/src/models/qwen4exp.cpp
|
||||
+++ b/src/models/qwen4exp.cpp
|
||||
@@ -560,9 +565,13 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
|
||||
ggml_build_forward_expand(gf, mctx_cur->cpy_k(ctx0, k_cur, k_idxs, il));
|
||||
ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));
|
||||
}
|
||||
|
||||
- ggml_tensor * kq_mask = inp->get_kq_mask();
|
||||
+ // get_kq_mask() returns the F16-converted mask when flash attention is
|
||||
+ // on, but ggml_fill/set_rows require F32. Mirror the MLA sparse path
|
||||
+ // (llm_graph_context::build_attn): run the top-k mask surgery on the F32
|
||||
+ // base mask and cast to the attention type only at the end.
|
||||
+ ggml_tensor * kq_mask = inp->self_kq_mask;
|
||||
|
||||
// prepare new kq mask - starts filled with -INFINITY
|
||||
ggml_tensor * kq_mask_all = ggml_fill(ctx0, kq_mask, -INFINITY);
|
||||
|
||||
@@ -592,9 +601,12 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
|
||||
ggml_tensor * q = q_cur;
|
||||
ggml_tensor * k = mctx_cur->get_k(ctx0, il);
|
||||
ggml_tensor * v = mctx_cur->get_v(ctx0, il);
|
||||
|
||||
- ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_top_k, nullptr, nullptr, kq_scale, il);
|
||||
+ // flash attention consumes an F16 mask; softmax consumes F32 as-is
|
||||
+ ggml_tensor * kq_mask_attn = inp->cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k, GGML_TYPE_F16) : kq_mask_top_k;
|
||||
+
|
||||
+ ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_attn, nullptr, nullptr, kq_scale, il);
|
||||
cb(cur, "kqv_out", il);
|
||||
|
||||
return cur;
|
||||
}
|
||||
@@ -59,7 +59,7 @@ fi
|
||||
: "${SPLIT_MODE:=none}"
|
||||
: "${FIT:=off}"
|
||||
: "${NGL:=999}"
|
||||
: "${FLASH_ATTN:=off}"
|
||||
: "${FLASH_ATTN:=on}"
|
||||
: "${TEMPERATURE:=0.7}"
|
||||
: "${TOP_K:=20}"
|
||||
: "${TOP_P:=0.8}"
|
||||
|
||||
@@ -88,7 +88,7 @@ llama-server \
|
||||
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
|
||||
--host 0.0.0.0 --port 7770 --jinja \
|
||||
--device ROCm0 --gpu-layers 999 \
|
||||
--split-mode none --fit off --flash-attn off \
|
||||
--split-mode none --fit off --flash-attn on \
|
||||
--ctx-size 32768 --parallel 1 \
|
||||
--batch-size 4096 --ubatch-size 4096 \
|
||||
--cache-type-k f16 --cache-type-v f16 \
|
||||
@@ -114,11 +114,18 @@ Key choices and deviations from the CIRU deployment:
|
||||
`GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
|
||||
runtime-only and are gone.
|
||||
- **`--mmproj` active** — vision tower included (f16, lossless).
|
||||
- **`--flash-attn off` is mandatory.** With fa on, the QSA indexer path
|
||||
(layers with attention compression) calls `ggml_fill` on the F16-cast kq
|
||||
mask and aborts graph build (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
|
||||
ggml.c:5371, backtrace through `build_attn_qsa`). Verified live on this
|
||||
v2 main — loads and serves fine with fa off.
|
||||
- **`--flash-attn on` (restored 2026-08-29).** fa on previously aborted graph
|
||||
build: the QSA indexer path (layers with attention compression) called
|
||||
`ggml_fill` on the F16-cast kq mask (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
|
||||
ggml.c:5371, backtrace through `build_attn_qsa`). Fixed in the fork source
|
||||
by the QSA F32-mask patch — vendored at
|
||||
`scripts/patches/qwen4exp-qsa-f32-mask.patch` (sha256
|
||||
`f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f`);
|
||||
the binary at `build-qwen38/bin/llama-server` (rebuilt 2026-08-29) already
|
||||
carries it. A clean rebuild must re-apply
|
||||
`git apply scripts/patches/qwen4exp-qsa-f32-mask.patch` inside
|
||||
`/home/dank/GitPackages/ROCmFPX` before `make`. Verified live at
|
||||
CTX_SIZE=32768: model loads, health 200, real completion.
|
||||
- Context capped at **32768 × 1 slot — the author's own quick-start**. This
|
||||
model is fully GPU-resident (87.06 GiB): the PLE n-gram table sits inside
|
||||
the GGUF as one tensor, unlike STRIX_LEAN which streamed it from RAM.
|
||||
@@ -158,6 +165,8 @@ best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
|
||||
- Env: `scripts/frank-default-large.env`
|
||||
- Serve: `scripts/serve-frank-large.sh`
|
||||
- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
|
||||
- Fork patch: `scripts/patches/qwen4exp-qsa-f32-mask.patch` (re-apply after
|
||||
any clean fork rebuild — see Serving profile)
|
||||
|
||||
## Swap procedure
|
||||
|
||||
@@ -188,8 +197,9 @@ loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.
|
||||
requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; every
|
||||
Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries
|
||||
only `blk.48`. Spec decoding is ngram-mod only. See the MTP section.
|
||||
4. **`--flash-attn off` is mandatory.** fa on aborts graph build on the QSA
|
||||
indexer path (F32 `ggml_fill` on the F16 mask) — see Serving profile.
|
||||
4. **`--flash-attn on` (restored 2026-08-29).** fa on aborts graph build
|
||||
without the QSA F32-mask patch — see Serving profile (fix + sha256
|
||||
vendored there).
|
||||
5. **`SVM mapping failed` kernel spam** is a documented symptom, not root
|
||||
cause — investigate before blaming memory pressure.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user