---
name: llama-flash-attn-qsa-fix
description: Restore --flash-attn on for qwen4exp models when QSA indexer path aborts with GGML_ASSERT F32 error
---

## Problem
`--flash-attn on` aborts graph build for qwen4exp models:
```
GGML_ASSERT(a->type == GGML_TYPE_F32)
at ggml/src/ggml.c:5371, backtrace through build_attn_qsa
```
Root cause: `build_attn_qsa` calls `ggml_fill` on the F16-cast kq mask returned by `get_kq_mask()` when flash attn is enabled, but `ggml_fill` requires F32.

## Solution
Apply the QSA F32-mask patch that mirrors the MLA sparse path (`llm_graph_context::build_attn`):
1. Use the F32 base mask (`inp->self_kq_mask`) instead of the F16-cast version for the top-k surgery
2. Cast to F16 only at the attention call when `flash_attn` is enabled

### Patch hunks (unified diff format)
```diff
@@ -560,9 +565,13 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
         ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));
     }
-    ggml_tensor * kq_mask = inp->get_kq_mask();
+    // get_kq_mask() returns the F16-converted mask when flash attention is
+    // on, but ggml_fill/set_rows require F32. Mirror the MLA sparse path
+    // (llm_graph_context::build_attn): run the top-k mask surgery on the F32
+    // base mask and cast to the attention type only at the end.
+    ggml_tensor * kq_mask = inp->self_kq_mask;
     // prepare new kq mask - starts filled with -INFINITY
     ggml_tensor * kq_mask_all = ggml_fill(ctx0, kq_mask, -INFINITY);
 
@@ -592,9 +601,12 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
     ggml_tensor * q = q_cur;
     ggml_tensor * k = mctx_cur->get_k(ctx0, il);
     ggml_tensor * v = mctx_cur->get_v(ctx0, il);
-    ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_top_k, nullptr, nullptr, kq_scale, il);
+    // flash attention consumes an F16 mask; softmax consumes F32 as-is
+    ggml_tensor * kq_mask_attn = inp->cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k, GGML_TYPE_F16) : kq_mask_top_k;
+
+    ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_attn, nullptr, nullptr, kq_scale, il);
     cb(cur, "kqv_out", il);
     return cur;
 }
```

## Procedure
1. **Apply patch to fork:**
   ```bash
   cd /home/dank/GitPackages/ROCmFPX
   git apply /path/to/qwen4exp-qsa-f32-mask.patch
   make -j16 llama-server
   ```
2. **Validate live (on test port):**
   ```bash
   sudo systemctl stop frank-default-large
   nohup ./build-qwen38/bin/llama-server \n    --model ... --flash-attn on --ctx-size 32768 --fit off -t 16 -tb 16 \n    --port 7999 > /tmp/fa-test.log 2>&1 &
   sleep 90 && grep 'model loaded' /tmp/fa-test.log && curl http://127.0.0.1:7999/health
   ```
3. **Verify completion:** Send a real `/v1/chat/completions` request — if it completes without asserts, the fix works.
4. **Deploy production:** Commit config (FLASH_ATTN=on), push, pull on target, restart service.

## Verification
- Model loads without asserts
- Health endpoint returns 200
- Real completion succeeds (finish: "stop")
- Journal clean: no `GGML_ASSERT`, no `cudaMalloc`, no `missing tensor`
- Flash attention engaged (check server logs for "flash attention" or timing improvement vs off)

## Known Constraints
- Works only with charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged)
- Does NOT work with CIRU runtime, upstream llama.cpp, or stock binaries
- ctx-size cap remains 32768 for full-model GPU residency (87 GiB model + KV buffer OOMs beyond)