frank-default-large: re-enable --flash-attn on via QSA F32-mask patch
fa on aborted graph build: build_attn_qsa called ggml_fill on the F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the MLA sparse path in llm_graph_context::build_attn (F32 base mask for the top-k surgery, F16 cast only at the attention call, gated on cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch (sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with it and validated live at CTX_SIZE=32768 (health 200, real completion).
This commit is contained in:
1 parent
101b1f8be2
commit
7cb6eeb67a
5 files changed
+70
-21
No files matched your search
@@ -88,7 +88,7 @@ llama-server \
|
||||
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
|
||||
--host 0.0.0.0 --port 7770 --jinja \
|
||||
--device ROCm0 --gpu-layers 999 \
|
||||
--split-mode none --fit off --flash-attn off \
|
||||
--split-mode none --fit off --flash-attn on \
|
||||
--ctx-size 32768 --parallel 1 \
|
||||
--batch-size 4096 --ubatch-size 4096 \
|
||||
--cache-type-k f16 --cache-type-v f16 \
|
||||
@@ -114,11 +114,18 @@ Key choices and deviations from the CIRU deployment:
|
||||
`GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
|
||||
runtime-only and are gone.
|
||||
- **`--mmproj` active** — vision tower included (f16, lossless).
|
||||
- **`--flash-attn off` is mandatory.** With fa on, the QSA indexer path
|
||||
(layers with attention compression) calls `ggml_fill` on the F16-cast kq
|
||||
mask and aborts graph build (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
|
||||
ggml.c:5371, backtrace through `build_attn_qsa`). Verified live on this
|
||||
v2 main — loads and serves fine with fa off.
|
||||
- **`--flash-attn on` (restored 2026-08-29).** fa on previously aborted graph
|
||||
build: the QSA indexer path (layers with attention compression) called
|
||||
`ggml_fill` on the F16-cast kq mask (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
|
||||
ggml.c:5371, backtrace through `build_attn_qsa`). Fixed in the fork source
|
||||
by the QSA F32-mask patch — vendored at
|
||||
`scripts/patches/qwen4exp-qsa-f32-mask.patch` (sha256
|
||||
`f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f`);
|
||||
the binary at `build-qwen38/bin/llama-server` (rebuilt 2026-08-29) already
|
||||
carries it. A clean rebuild must re-apply
|
||||
`git apply scripts/patches/qwen4exp-qsa-f32-mask.patch` inside
|
||||
`/home/dank/GitPackages/ROCmFPX` before `make`. Verified live at
|
||||
CTX_SIZE=32768: model loads, health 200, real completion.
|
||||
- Context capped at **32768 × 1 slot — the author's own quick-start**. This
|
||||
model is fully GPU-resident (87.06 GiB): the PLE n-gram table sits inside
|
||||
the GGUF as one tensor, unlike STRIX_LEAN which streamed it from RAM.
|
||||
@@ -158,6 +165,8 @@ best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
|
||||
- Env: `scripts/frank-default-large.env`
|
||||
- Serve: `scripts/serve-frank-large.sh`
|
||||
- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
|
||||
- Fork patch: `scripts/patches/qwen4exp-qsa-f32-mask.patch` (re-apply after
|
||||
any clean fork rebuild — see Serving profile)
|
||||
|
||||
## Swap procedure
|
||||
|
||||
@@ -188,8 +197,9 @@ loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.
|
||||
requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; every
|
||||
Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries
|
||||
only `blk.48`. Spec decoding is ngram-mod only. See the MTP section.
|
||||
4. **`--flash-attn off` is mandatory.** fa on aborts graph build on the QSA
|
||||
indexer path (F32 `ggml_fill` on the F16 mask) — see Serving profile.
|
||||
4. **`--flash-attn on` (restored 2026-08-29).** fa on aborts graph build
|
||||
without the QSA F32-mask patch — see Serving profile (fix + sha256
|
||||
vendored there).
|
||||
5. **`SVM mapping failed` kernel spam** is a documented symptom, not root
|
||||
cause — investigate before blaming memory pressure.
|
||||
|
||||
|
||||
Reference in new issue
Block a user