Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
74 lines
3.4 KiB
Markdown
74 lines
3.4 KiB
Markdown
---
|
|
name: llama-flash-attn-qsa-fix
|
|
description: Restore --flash-attn on for qwen4exp models when QSA indexer path aborts with GGML_ASSERT F32 error
|
|
---
|
|
|
|
## Problem
|
|
`--flash-attn on` aborts graph build for qwen4exp models:
|
|
```
|
|
GGML_ASSERT(a->type == GGML_TYPE_F32)
|
|
at ggml/src/ggml.c:5371, backtrace through build_attn_qsa
|
|
```
|
|
Root cause: `build_attn_qsa` calls `ggml_fill` on the F16-cast kq mask returned by `get_kq_mask()` when flash attn is enabled, but `ggml_fill` requires F32.
|
|
|
|
## Solution
|
|
Apply the QSA F32-mask patch that mirrors the MLA sparse path (`llm_graph_context::build_attn`):
|
|
1. Use the F32 base mask (`inp->self_kq_mask`) instead of the F16-cast version for the top-k surgery
|
|
2. Cast to F16 only at the attention call when `flash_attn` is enabled
|
|
|
|
### Patch hunks (unified diff format)
|
|
```diff
|
|
@@ -560,9 +565,13 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
|
|
ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));
|
|
}
|
|
- ggml_tensor * kq_mask = inp->get_kq_mask();
|
|
+ // get_kq_mask() returns the F16-converted mask when flash attention is
|
|
+ // on, but ggml_fill/set_rows require F32. Mirror the MLA sparse path
|
|
+ // (llm_graph_context::build_attn): run the top-k mask surgery on the F32
|
|
+ // base mask and cast to the attention type only at the end.
|
|
+ ggml_tensor * kq_mask = inp->self_kq_mask;
|
|
// prepare new kq mask - starts filled with -INFINITY
|
|
ggml_tensor * kq_mask_all = ggml_fill(ctx0, kq_mask, -INFINITY);
|
|
|
|
@@ -592,9 +601,12 @@ ggml_tensor * llama_model_qwen4exp::graph::build_attn_qsa(
|
|
ggml_tensor * q = q_cur;
|
|
ggml_tensor * k = mctx_cur->get_k(ctx0, il);
|
|
ggml_tensor * v = mctx_cur->get_v(ctx0, il);
|
|
- ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_top_k, nullptr, nullptr, kq_scale, il);
|
|
+ // flash attention consumes an F16 mask; softmax consumes F32 as-is
|
|
+ ggml_tensor * kq_mask_attn = inp->cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k, GGML_TYPE_F16) : kq_mask_top_k;
|
|
+
|
|
+ ggml_tensor * cur = build_attn_mha(q, k, v, nullptr, kq_mask_attn, nullptr, nullptr, kq_scale, il);
|
|
cb(cur, "kqv_out", il);
|
|
return cur;
|
|
}
|
|
```
|
|
|
|
## Procedure
|
|
1. **Apply patch to fork:**
|
|
```bash
|
|
cd /home/dank/GitPackages/ROCmFPX
|
|
git apply /path/to/qwen4exp-qsa-f32-mask.patch
|
|
make -j16 llama-server
|
|
```
|
|
2. **Validate live (on test port):**
|
|
```bash
|
|
sudo systemctl stop frank-default-large
|
|
nohup ./build-qwen38/bin/llama-server \n --model ... --flash-attn on --ctx-size 32768 --fit off -t 16 -tb 16 \n --port 7999 > /tmp/fa-test.log 2>&1 &
|
|
sleep 90 && grep 'model loaded' /tmp/fa-test.log && curl http://127.0.0.1:7999/health
|
|
```
|
|
3. **Verify completion:** Send a real `/v1/chat/completions` request — if it completes without asserts, the fix works.
|
|
4. **Deploy production:** Commit config (FLASH_ATTN=on), push, pull on target, restart service.
|
|
|
|
## Verification
|
|
- Model loads without asserts
|
|
- Health endpoint returns 200
|
|
- Real completion succeeds (finish: "stop")
|
|
- Journal clean: no `GGML_ASSERT`, no `cudaMalloc`, no `missing tensor`
|
|
- Flash attention engaged (check server logs for "flash attention" or timing improvement vs off)
|
|
|
|
## Known Constraints
|
|
- Works only with charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged)
|
|
- Does NOT work with CIRU runtime, upstream llama.cpp, or stock binaries
|
|
- ctx-size cap remains 32768 for full-model GPU residency (87 GiB model + KV buffer OOMs beyond)
|