dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-large: verify v2 main on build-qwen38 — flash-attn off required, MTP draft incompatible (ngram-mod fallback)

Browse Source 
- Main model (v2 joined PLE layout, sha256 1e4160ca…) loads and serves on
  build-qwen38 only with --flash-attn off: the QSA indexer path calls
  ggml_fill on the F16-cast kq mask and aborts graph build (GGML_ASSERT
  F32 at ggml.c:5371). Verified live: model loaded, health 200, real
  completion.
- MTP draft head (agentionai FAST, 2.28 GiB) does not load on this binary:
  qwen4exp loader requires full per-layer blk.<il>.hc_attn_norm.weight,
  head carries only blk.48. Third confirmation (EasiiX Q8_0 + agentionai
  FAST vs kingjones STRIX_LEAN previously, now own-matched pair). Spec
  decoding runs ngram-mod only; head retained for a future compatible build.
- FLASH_ATTN=off and SPEC_TYPE=ngram-mod wired through env + serve script;
  runbook and AGENTS.md updated with the verified constraints.
This commit is contained in:
Frank
2026-08-29 21:39:37 -07:00
parent 3c2a5d2fe7
commit 9865f9e307
4 changed files with 129 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md
+67 -39
View File
@@ -1,7 +1,7 @@
---
type: reference
title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix, per-head PLE in-GGUF, own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large.
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large.
tags:
- llama.cpp
- rocm
@@ -17,29 +17,33 @@ timestamp: 2026-08-29
# agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
> **Active since 2026-08-29.** Replaced the jcbtc CIRU v1.1 (ciru-ai runtime)
> deployment on the same slot.
> **Active since 2026-08-29.** Replaced the kingjones STRIX_LEAN and jcbtc
> CIRU v1.1 deployments on the same slot.
> **MTP draft disabled** — the author's own head does not load on
> build-qwen38 (see Known issues #3). Speculative decoding is ngram-mod only.
`frank-default-large` (port 7770) serves
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf`) with the author's
own MTP draft head (`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout)
with the author's own MTP draft head
(`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the
charlie12345/ROCmFPX `build-qwen38` runtime.
## Why this model
- 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory
box, including the 51.2 B-parameter n-gram (PLE) table, which is split per
head inside the single GGUF (`v2/` is the joined-table layout; root is the
VRAM-resident one and is what we run).
box, including the 51.2 B-parameter n-gram (PLE) table. REPO `v2/` layout
is used (joined single `per_layer_token_embd.weight` tensor) because the
charlie12345/ROCmFPX loader requires it; the root `-ple16` file's per-head
split only loads in the LaurentZuijdwijk Vulkan fork (see Known issues #2).
- imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2
(better than AesSedai IQ4_XS / IQ3_S at smaller size).
- 18-25% faster prompt processing than a mainline quant of similar size
(356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth).
- Ships the model's own f16 vision tower (`mmproj/`) and its own MTP head
(~66% draft acceptance, mean accepted length 2.97 of 3 at n-max 3).
- Ships the model's own f16 vision tower (`mmproj/`).
- The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0
draft measured worse on this head family (acceptance + throughput).
draft measured worse on this head family — but it does not load on
build-qwen38 (Known issues #3).
## Runtime
@@ -61,7 +65,8 @@ author's own matched MTP head on this binary.
| Item | Path on Dankclaw | Size |
|---|---|---|
| Main | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) |
| Main (v2 joined) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf` | 93,484,242,816 B (87.06 GiB) |
| Main (root ple16, unused) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) |
| Vision | `…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf` | 904,004,128 B |
| Draft head | `/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf` | 2,444,519,296 B |
@@ -79,11 +84,11 @@ build-qwen38 profile:
```
llama-server \
--model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf \
--model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf \
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
--host 0.0.0.0 --port 7770 --jinja \
--device ROCm0 --gpu-layers 999 \
--split-mode none --fit off --flash-attn on \
--split-mode none --fit off --flash-attn off \
--ctx-size 262144 --parallel 1 \
--batch-size 4096 --ubatch-size 4096 \
--cache-type-k f16 --cache-type-v f16 \
@@ -94,11 +99,7 @@ llama-server \
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
--chat-template-file chat-template.jinja \
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
--spec-type draft-mtp,ngram-mod \
--spec-draft-model Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf \
--spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.75 \
--spec-draft-ngl 999 --spec-draft-device ROCm0 \
--spec-draft-type-k f16 --spec-draft-type-v f16
--spec-type ngram-mod
```
Key choices and deviations from the CIRU deployment:
@@ -107,27 +108,45 @@ Key choices and deviations from the CIRU deployment:
(same as every Flash-Next deployment).
- **`--checkpoint-min-step` removed** — build-qwen38 does not implement it
(verified via `--help`); only `--ctx-checkpoints` is passed.
- **No PLE sidecar args** — the n-gram table is in-GGUF (per-head split),
not a `ple/` CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the
- **No PLE sidecar args** — the n-gram table is in-GGUF (joined
`per_layer_token_embd.weight` from the `v2/` layout), not a `ple/`
CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the
`GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
runtime-only and are gone.
- **`--mmproj` active** — vision tower included (f16, lossless).
- **MTP n-max 3, no p-split** — author measurements: n-max 3 = 32.4 t/s /
0.612 acceptance, adaptive p-split measured worse (28.4 t/s / 0.468).
`p-min` stays at the 0.75 default.
- **`--flash-attn off` is mandatory.** With fa on, the QSA indexer path
(layers with attention compression) calls `ggml_fill` on the F16-cast kq
mask and aborts graph build (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
ggml.c:5371, backtrace through `build_attn_qsa`). Verified live on this
v2 main — loads and serves fine with fa off.
- Context kept at 262144 × 1 slot (same as the CIRU deployment; the author's
quick-start is 32768 but the full window is fine on SVM).
- MTP is disabled (ngram-mod only) — see the MTP section below.
## MTP draft head
## MTP draft head — INCOMPATIBLE on build-qwen38
The draft is agentionai's own head for this exact model:
`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, architecture
`qwen4exp` with `blk.48.hc_attn_norm.weight` present (count 1, same shape as
the CIRU draft that works). The earlier 2026-08-28 failure of this head on
the *kingjones* STRIX_LEAN main model (`missing tensor blk.0.hc_attn_norm`)
was a cross-vendor pairing problem, not a defect of the head itself. The
draft carries a 512-expert MoE, so adaptive/long drafting costs real forward
passes — keep n-max 3.
The author's own matched head (`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-
FAST-GGUF`, 2.28 GiB, `qwen4exp` arch) does **not** load on this binary:
```
error loading model: missing tensor 'blk.0.hc_attn_norm.weight'
```
The build-qwen38 qwen4exp loader requires a full per-layer
`blk.<il>.hc_attn_norm.weight` set in the draft (the graph reads all 48
layers); the head only carries `blk.48.hc_attn_norm.weight` (count 1 — all
three Flash-Next-era heads examined on this fork have that same partial
layout). Verified three times, not a cross-vendor artifact:
1. EasiiX Q8_0 head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
2. agentionai FAST head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
3. agentionai FAST head vs agentionai v2 main (2026-08-29, this deployment)
— failed identically.
The 2.28 GiB head stays on disk; the moment a binary or head appears that
carries the full per-layer norm set, re-enable with
`SPEC_TYPE=draft-mtp,ngram-mod`, `--spec-draft-n-max 3` (author-measured
best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
## Config files
@@ -152,12 +171,21 @@ loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.
1. **ROCmFP4 types require the ROCmFPX fork.** Stock llama.cpp and the CIRU
runtime reject these files.
2. **Single 87 GiB file.** Use resumable curl; allow ~10-20 minutes on a
good link.
3. **MTP head must be agentionai's.** EasiiX Q8_0 and other heads are
incompatible with this binary's draft expectations for Flash-Next
(hyper-connection norm tensor).
4. **`SVM mapping failed` kernel spam** is a documented symptom, not root
2. **Use the `v2/` joined layout, NOT the root `-ple16` split.** This binary's
qwen4exp loader (qwen4exp.cpp:104-108) unconditionally requires a single
`per_layer_token_embd.weight` tensor and GGML_ASSERTs "qwen4exp is missing
the PLE n-gram table" against the per-head split. The root file only loads
in the LaurentZuijdwijk Vulkan fork (`vulkan-qwen4exp-rocmfpx`). The v2
file is byte-identical in weights, ~5 KB larger header, loads fully into
SVM unified memory (no `--ngram-on-disk` needed — kingjones STRIX ran a
50 GiB joined PLE table the same way).
3. **No MTP draft head loads on build-qwen38.** The qwen4exp loader
requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; every
Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries
only `blk.48`. Spec decoding is ngram-mod only. See the MTP section.
4. **`--flash-attn off` is mandatory.** fa on aborts graph build on the QSA
indexer path (F32 `ggml_fill` on the F16 mask) — see Serving profile.
5. **`SVM mapping failed` kernel spam** is a documented symptom, not root
cause — investigate before blaming memory pressure.
## Citations