frank-default-large: re-enable --flash-attn on via QSA F32-mask patch

fa on aborted graph build: build_attn_qsa called ggml_fill on the
F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the
MLA sparse path in llm_graph_context::build_attn (F32 base mask for the
top-k surgery, F16 cast only at the attention call, gated on
cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch
(sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with
it and validated live at CTX_SIZE=32768 (health 200, real completion).
This commit is contained in:
Frank committed 2026-08-29 23:43:25 -07:00
1 parent 101b1f8be2
commit 7cb6eeb67a
5 files changed
+70 -21

No files matched your search

@@ -88,7 +88,7 @@ llama-server \
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
--host 0.0.0.0 --port 7770 --jinja \
--device ROCm0 --gpu-layers 999 \
--split-mode none --fit off --flash-attn off \
--split-mode none --fit off --flash-attn on \
--ctx-size 32768 --parallel 1 \
--batch-size 4096 --ubatch-size 4096 \
--cache-type-k f16 --cache-type-v f16 \
@@ -114,11 +114,18 @@ Key choices and deviations from the CIRU deployment:
`GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
runtime-only and are gone.
- **`--mmproj` active** — vision tower included (f16, lossless).
- **`--flash-attn off` is mandatory.** With fa on, the QSA indexer path
(layers with attention compression) calls `ggml_fill` on the F16-cast kq
mask and aborts graph build (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
ggml.c:5371, backtrace through `build_attn_qsa`). Verified live on this
v2 main — loads and serves fine with fa off.
- **`--flash-attn on` (restored 2026-08-29).** fa on previously aborted graph
build: the QSA indexer path (layers with attention compression) called
`ggml_fill` on the F16-cast kq mask (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
ggml.c:5371, backtrace through `build_attn_qsa`). Fixed in the fork source
by the QSA F32-mask patch — vendored at
`scripts/patches/qwen4exp-qsa-f32-mask.patch` (sha256
`f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f`);
the binary at `build-qwen38/bin/llama-server` (rebuilt 2026-08-29) already
carries it. A clean rebuild must re-apply
`git apply scripts/patches/qwen4exp-qsa-f32-mask.patch` inside
`/home/dank/GitPackages/ROCmFPX` before `make`. Verified live at
CTX_SIZE=32768: model loads, health 200, real completion.
- Context capped at **32768 × 1 slot — the author's own quick-start**. This
model is fully GPU-resident (87.06 GiB): the PLE n-gram table sits inside
the GGUF as one tensor, unlike STRIX_LEAN which streamed it from RAM.
@@ -158,6 +165,8 @@ best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
- Env: `scripts/frank-default-large.env`
- Serve: `scripts/serve-frank-large.sh`
- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
- Fork patch: `scripts/patches/qwen4exp-qsa-f32-mask.patch` (re-apply after
any clean fork rebuild — see Serving profile)
## Swap procedure
@@ -188,8 +197,9 @@ loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.
requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; every
Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries
only `blk.48`. Spec decoding is ngram-mod only. See the MTP section.
4. **`--flash-attn off` is mandatory.** fa on aborts graph build on the QSA
indexer path (F32 `ggml_fill` on the F16 mask) — see Serving profile.
4. **`--flash-attn on` (restored 2026-08-29).** fa on aborts graph build
without the QSA F32-mask patch — see Serving profile (fix + sha256
vendored there).
5. **`SVM mapping failed` kernel spam** is a documented symptom, not root
cause — investigate before blaming memory pressure.