command-center/agent/managed-skills/llama-flash-attn-qsa-fix/SKILL.md

name, description

name	description
llama-flash-attn-qsa-fix	Restore --flash-attn on for qwen4exp models when QSA indexer path aborts with GGML_ASSERT F32 error

Problem

--flash-attn on aborts graph build for qwen4exp models:

GGML_ASSERT(a->type == GGML_TYPE_F32)
at ggml/src/ggml.c:5371, backtrace through build_attn_qsa

Root cause: build_attn_qsa calls ggml_fill on the F16-cast kq mask returned by get_kq_mask() when flash attn is enabled, but ggml_fill requires F32.

Solution

Apply the QSA F32-mask patch that mirrors the MLA sparse path (llm_graph_context::build_attn):

1. Use the F32 base mask (inp->self_kq_mask) instead of the F16-cast version for the top-k surgery
2. Cast to F16 only at the attention call when flash_attn is enabled

Patch hunks (unified diff format)

@@ -560,9 +565,13 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
         ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));
     }
-    ggml_tensor * kq_mask = inp->get_kq_mask();
+    // get_kq_mask() returns the F16-converted mask when flash attention is
+    // on, but ggml_fill/set_rows require F32. Mirror the MLA sparse path
+    // (llm_graph_context::build_attn): run the top-k mask surgery on the F32
+    // base mask and cast to the attention type only at the end.
+    ggml_tensor * kq_mask = inp->self_kq_mask;
     // prepare new kq mask - starts filled with -INFINITY
     ggml_tensor * kq_mask_all = ggml_fill(ctx0, kq_mask, -INFINITY);
 
@@ -592,9 +601,12 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
     ggml_tensor * q = q_cur;
     ggml_tensor * k = mctx_cur->get_k(ctx0, il);
     ggml_tensor * v = mctx_cur->get_v(ctx0, il);
-    ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_top_k, nullptr, nullptr, kq_scale, il);
+    // flash attention consumes an F16 mask; softmax consumes F32 as-is
+    ggml_tensor * kq_mask_attn = inp->cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k, GGML_TYPE_F16) : kq_mask_top_k;
+
+    ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_attn, nullptr, nullptr, kq_scale, il);
     cb(cur, "kqv_out", il);
     return cur;
 }

Procedure

1. Apply patch to fork:

   cd /home/dank/GitPackages/ROCmFPX
   git apply /path/to/qwen4exp-qsa-f32-mask.patch
   make -j16 llama-server
   

2. Validate live (on test port):

   sudo systemctl stop frank-default-large
   nohup ./build-qwen38/bin/llama-server \n    --model ... --flash-attn on --ctx-size 32768 --fit off -t 16 -tb 16 \n    --port 7999 > /tmp/fa-test.log 2>&1 &
   sleep 90 && grep 'model loaded' /tmp/fa-test.log && curl http://127.0.0.1:7999/health
   

3. Verify completion: Send a real /v1/chat/completions request — if it completes without asserts, the fix works.
4. Deploy production: Commit config (FLASH_ATTN=on), push, pull on target, restart service.

Verification

• Model loads without asserts
• Health endpoint returns 200
• Real completion succeeds (finish: "stop")
• Journal clean: no GGML_ASSERT, no cudaMalloc, no missing tensor
• Flash attention engaged (check server logs for "flash attention" or timing improvement vs off)

Known Constraints

• Works only with charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged)
• Does NOT work with CIRU runtime, upstream llama.cpp, or stock binaries
• ctx-size cap remains 32768 for full-model GPU residency (87 GiB model + KV buffer OOMs beyond)