From 7cb6eeb67ae5af6f6b3a533920c0072db06618e4 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 29 Aug 2026 23:43:25 -0700 Subject: [PATCH] frank-default-large: re-enable --flash-attn on via QSA F32-mask patch fa on aborted graph build: build_attn_qsa called ggml_fill on the F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the MLA sparse path in llm_graph_context::build_attn (F32 base mask for the top-k surgery, F16 cast only at the attention call, gated on cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch (sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with it and validated live at CTX_SIZE=32768 (health 200, real completion). --- AGENTS.md | 12 ++++--- scripts/frank-default-large.env | 20 +++++++----- scripts/patches/qwen4exp-qsa-f32-mask.patch | 31 +++++++++++++++++++ scripts/serve-frank-large.sh | 2 +- .../LLM/agentionai-fast-imatrix.md | 26 +++++++++++----- 5 files changed, 70 insertions(+), 21 deletions(-) create mode 100644 scripts/patches/qwen4exp-qsa-f32-mask.patch diff --git a/AGENTS.md b/AGENTS.md index a231215..fd9483a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -298,9 +298,10 @@ slot (author quick-start; the fully GPU-resident 87 GiB model + f16 KV + compute buffer OOMs the SVM pool beyond that — verified live: cudaMalloc fails at 131072/262144), b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, `--fit off` (auto-fit segfaults on qwen4exp hyper-connection config), -`--flash-attn off` (required: fa on aborts graph build on the QSA indexer -path — F32 `ggml_fill` on the F16 kq mask, ggml.c:5371), and reasoning on (Sharp -template, budget -1, `--reasoning-format deepseek`; chat kwargs keep +`--flash-attn on` (restored 2026-08-29 via the QSA F32-mask patch, +`scripts/patches/qwen4exp-qsa-f32-mask.patch` — re-apply inside +`/home/dank/GitPackages/ROCmFPX` before any clean rebuild), and reasoning on +(Sharp template, budget -1, `--reasoning-format deepseek`; chat kwargs keep `preserve_thinking` + `preserve_reasoning`). Env `scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`. @@ -309,7 +310,10 @@ model (v2 joined PLE layout), replacing the kingjones STRIX and jcbtc CIRU v1.1 deployments. The own-matched MTP draft head does NOT load on build-qwen38 — verified against the v2 main itself (`missing tensor blk.0.hc_attn_norm.weight`), third confirmation (EasiiX Q8_0 + agentionai FAST vs STRIX_LEAN earlier). The -head stays on disk for a future compatible binary. +head stays on disk for a future compatible binary. Same day, `--flash-attn on` +was restored: the QSA F32-mask patch (vendored +`scripts/patches/qwen4exp-qsa-f32-mask.patch`) fixes the F16-kq_mask +`ggml_fill` abort; the rebuilt fork binary carries it, verified live at 32K. Runbook: [serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md). diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index bbaa99c..6ba6896 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -10,12 +10,16 @@ # # Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used, # 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE -# KNOWN CONSTRAINTS on the charlie12345/ROCmFPX build-qwen38 runtime. -# 1. FA must stay OFF. --flash-attn on aborts graph build: the QSA -# indexer path calls ggml_fill on the F16-cast kq mask (ggml_fill -# asserts GGML_TYPE_F32 at ggml.c:5371). The v2 main loads and serves -# fine with --flash-attn off (verified live: model loaded, health 200, -# real completion). +# 1. Flash attention is ON. fa on used to abort graph build: the QSA +# indexer path called ggml_fill on the F16-cast kq mask (ggml_fill +# asserts GGML_TYPE_F32 at ggml.c:5371). Fixed in the fork source by +# the QSA F32-mask patch (scripts/patches/qwen4exp-qsa-f32-mask.patch, +# sha256 f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f) +# and the rebuilt binary at LLAMA_BIN already carries the fix. A clean +# rebuild must re-apply: git apply scripts/patches/qwen4exp-qsa-f32-mask.patch +# inside /home/dank/GitPackages/ROCmFPX before make. Verified live: +# --flash-attn on + v2 main at CTX_SIZE=32768 loads, health 200, real +# completion. # 2. The MTP draft head does NOT load on this binary. The qwen4exp loader # requires blk.0.hc_attn_norm.weight per layer; the agentionai FAST # head carries only blk.48 (verified 3x: EasiiX Q8_0 + FAST against @@ -54,7 +58,7 @@ ALIAS=frank-default-large BIND_ADDR=0.0.0.0 LISTEN_PORT=7770 -# Serving: 131072-token context on ONE slot (parallel 1), batch 4096 / +# Serving: 32768-token context on ONE slot (parallel 1), batch 4096 / # ubatch 4096, 16+16 threads, f16 KV, 16 context checkpoints, 8 GiB # prompt cache. --fit off: auto-fit segfaults on this model's # hyper-connection/indexer config; -ngl 999 puts every layer on the @@ -81,7 +85,7 @@ SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state SPLIT_MODE=none FIT=off NGL=999 -FLASH_ATTN=off +FLASH_ATTN=on # Sampling (card instruct table, non-thinking). TEMPERATURE=0.7 diff --git a/scripts/patches/qwen4exp-qsa-f32-mask.patch b/scripts/patches/qwen4exp-qsa-f32-mask.patch new file mode 100644 index 0000000..ac1af3e --- /dev/null +++ b/scripts/patches/qwen4exp-qsa-f32-mask.patch @@ -0,0 +1,31 @@ +--- a/src/models/qwen4exp.cpp ++++ b/src/models/qwen4exp.cpp +@@ -560,9 +565,13 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa( + ggml_build_forward_expand(gf, mctx_cur->cpy_k(ctx0, k_cur, k_idxs, il)); + ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il)); + } + +- ggml_tensor * kq_mask = inp->get_kq_mask(); ++ // get_kq_mask() returns the F16-converted mask when flash attention is ++ // on, but ggml_fill/set_rows require F32. Mirror the MLA sparse path ++ // (llm_graph_context::build_attn): run the top-k mask surgery on the F32 ++ // base mask and cast to the attention type only at the end. ++ ggml_tensor * kq_mask = inp->self_kq_mask; + + // prepare new kq mask - starts filled with -INFINITY + ggml_tensor * kq_mask_all = ggml_fill(ctx0, kq_mask, -INFINITY); + +@@ -592,9 +601,12 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa( + ggml_tensor * q = q_cur; + ggml_tensor * k = mctx_cur->get_k(ctx0, il); + ggml_tensor * v = mctx_cur->get_v(ctx0, il); + +- ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_top_k, nullptr, nullptr, kq_scale, il); ++ // flash attention consumes an F16 mask; softmax consumes F32 as-is ++ ggml_tensor * kq_mask_attn = inp->cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k, GGML_TYPE_F16) : kq_mask_top_k; ++ ++ ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_attn, nullptr, nullptr, kq_scale, il); + cb(cur, "kqv_out", il); + + return cur; + } diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index a394349..a136c1d 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -59,7 +59,7 @@ fi : "${SPLIT_MODE:=none}" : "${FIT:=off}" : "${NGL:=999}" -: "${FLASH_ATTN:=off}" +: "${FLASH_ATTN:=on}" : "${TEMPERATURE:=0.7}" : "${TOP_K:=20}" : "${TOP_P:=0.8}" diff --git a/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md b/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md index 2487312..03a57b1 100644 --- a/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md +++ b/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md @@ -88,7 +88,7 @@ llama-server \ --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \ --host 0.0.0.0 --port 7770 --jinja \ --device ROCm0 --gpu-layers 999 \ - --split-mode none --fit off --flash-attn off \ + --split-mode none --fit off --flash-attn on \ --ctx-size 32768 --parallel 1 \ --batch-size 4096 --ubatch-size 4096 \ --cache-type-k f16 --cache-type-v f16 \ @@ -114,11 +114,18 @@ Key choices and deviations from the CIRU deployment: `GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU- runtime-only and are gone. - **`--mmproj` active** — vision tower included (f16, lossless). -- **`--flash-attn off` is mandatory.** With fa on, the QSA indexer path - (layers with attention compression) calls `ggml_fill` on the F16-cast kq - mask and aborts graph build (`GGML_ASSERT(a->type == GGML_TYPE_F32)`, - ggml.c:5371, backtrace through `build_attn_qsa`). Verified live on this - v2 main — loads and serves fine with fa off. +- **`--flash-attn on` (restored 2026-08-29).** fa on previously aborted graph + build: the QSA indexer path (layers with attention compression) called + `ggml_fill` on the F16-cast kq mask (`GGML_ASSERT(a->type == GGML_TYPE_F32)`, + ggml.c:5371, backtrace through `build_attn_qsa`). Fixed in the fork source + by the QSA F32-mask patch — vendored at + `scripts/patches/qwen4exp-qsa-f32-mask.patch` (sha256 + `f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f`); + the binary at `build-qwen38/bin/llama-server` (rebuilt 2026-08-29) already + carries it. A clean rebuild must re-apply + `git apply scripts/patches/qwen4exp-qsa-f32-mask.patch` inside + `/home/dank/GitPackages/ROCmFPX` before `make`. Verified live at + CTX_SIZE=32768: model loads, health 200, real completion. - Context capped at **32768 × 1 slot — the author's own quick-start**. This model is fully GPU-resident (87.06 GiB): the PLE n-gram table sits inside the GGUF as one tensor, unlike STRIX_LEAN which streamed it from RAM. @@ -158,6 +165,8 @@ best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse). - Env: `scripts/frank-default-large.env` - Serve: `scripts/serve-frank-large.sh` - Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp) +- Fork patch: `scripts/patches/qwen4exp-qsa-f32-mask.patch` (re-apply after + any clean fork rebuild — see Serving profile) ## Swap procedure @@ -188,8 +197,9 @@ loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it. requires a full per-layer `blk..hc_attn_norm.weight` set; every Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries only `blk.48`. Spec decoding is ngram-mod only. See the MTP section. -4. **`--flash-attn off` is mandatory.** fa on aborts graph build on the QSA - indexer path (F32 `ggml_fill` on the F16 mask) — see Serving profile. +4. **`--flash-attn on` (restored 2026-08-29).** fa on aborts graph build + without the QSA F32-mask patch — see Serving profile (fix + sha256 + vendored there). 5. **`SVM mapping failed` kernel spam** is a documented symptom, not root cause — investigate before blaming memory pressure.