dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-large: verify v2 main on build-qwen38 — flash-attn off required, MTP draft incompatible (ngram-mod fallback)

Browse Source 
- Main model (v2 joined PLE layout, sha256 1e4160ca…) loads and serves on
  build-qwen38 only with --flash-attn off: the QSA indexer path calls
  ggml_fill on the F16-cast kq mask and aborts graph build (GGML_ASSERT
  F32 at ggml.c:5371). Verified live: model loaded, health 200, real
  completion.
- MTP draft head (agentionai FAST, 2.28 GiB) does not load on this binary:
  qwen4exp loader requires full per-layer blk.<il>.hc_attn_norm.weight,
  head carries only blk.48. Third confirmation (EasiiX Q8_0 + agentionai
  FAST vs kingjones STRIX_LEAN previously, now own-matched pair). Spec
  decoding runs ngram-mod only; head retained for a future compatible build.
- FLASH_ATTN=off and SPEC_TYPE=ngram-mod wired through env + serve script;
  runbook and AGENTS.md updated with the verified constraints.
This commit is contained in:
Frank
2026-08-29 21:39:37 -07:00
parent 3c2a5d2fe7
commit 9865f9e307
4 changed files with 129 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+17 -11
View File
@@ -27,7 +27,7 @@ host.
| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head (ROCmFP4-FAST), 262K context / 1 slot, reasoning on | 7770 |
| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, PLE joined table in v2 GGUF), ngram-mod spec (MTP head on disk, incompatible with this build), 262K context / 1 slot, reasoning on | 7770 |
History: the 2026-08-17 redo retired the pipeline-parallel split and the
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
@@ -285,23 +285,29 @@ charlie12345/ROCmFPX `llama-server` built at
merged). The model is a 56B MoE (512 experts, 10 used), 48 layers (12 attn +
36 SSM, full_attention_interval=4), QSA indexer (4 heads, top_k=2048),
hyper-connections (count=4, low_rank=320), PLE (layer 1). The 51.2 B-parameter
n-gram table is split per head INSIDE the single GGUF — fully VRAM-resident,
no sidecar, no `--ngram-on-disk`. The model's own MTP draft head
n-gram table is a single joined tensor in the `v2/` GGUF we serve (the root
`-ple16` per-head split requires the LaurentZuijdwijk Vulkan fork and aborts
this loader's `GGML_ASSERT(qwen4exp is missing the PLE n-gram table)`) — fully
VRAM-resident, no sidecar, no `--ngram-on-disk`. The model's own MTP draft head
(`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, ROCmFP4-FAST
quant matched to target, ~66% draft acceptance) drives speculative decoding
at depth 3; vision tower (f16) ships as mmproj/. It runs 262144 context / 1
quant matched to target) is on disk but does **not** load on this binary
(loader requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; the head
carries only `blk.48`) — speculative decoding runs ngram-mod only. Vision
tower (f16) ships as mmproj/. It runs 262144 context / 1
slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, `--fit off` (auto-fit
segfaults on qwen4exp hyper-connection config), and reasoning on (Sharp
segfaults on qwen4exp hyper-connection config), `--flash-attn off` (required:
fa on aborts graph build on the QSA indexer path — F32 `ggml_fill` on the F16
kq mask, ggml.c:5371), and reasoning on (Sharp
template, budget -1, `--reasoning-format deepseek`; chat kwargs keep
`preserve_thinking` + `preserve_reasoning`). Env
`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`.
**Serving status (2026-08-29):** deployed the agentionai ROCmFP4-FAST imatrix
model with its own matched MTP draft head, replacing the jcbtc CIRU v1.1
deployment (ciru-ai runtime). The 2026-08-28 failure of this head on the
kingjones STRIX_LEAN model (`missing tensor blk.0.hc_attn_norm.weight`) was a
cross-vendor pairing problem — the head carries the hyper-connection norm
tensor (`blk.48.hc_attn_norm.weight`) and works against its own main file.
model (v2 joined PLE layout), replacing the kingjones STRIX and jcbtc CIRU v1.1
deployments. The own-matched MTP draft head does NOT load on build-qwen38 —
verified against the v2 main itself (`missing tensor blk.0.hc_attn_norm.weight`),
third confirmation (EasiiX Q8_0 + agentionai FAST vs STRIX_LEAN earlier). The
head stays on disk for a future compatible binary.
Runbook:
[serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md).
scripts/frank-default-large.env
+31 -32
View File
@@ -10,25 +10,20 @@
#
# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
# at layer 1) quantized Q4_0_ROCMFP4_FAST imatrix-calibrated, 87.06 GiB,
# 4.23 bpw. The n-gram PLE table is split per head inside the single GGUF
# (fully VRAM-resident — no ple sidecar, no --ngram-on-disk). Vision tower
# (f16, lossless) ships as mmproj/. The model's own MTP head (2.28 GiB,
# matched ROCmFP4-FAST quant, ~66% draft acceptance) drives spec decoding.
# KNOWN CONSTRAINTS on the charlie12345/ROCmFPX build-qwen38 runtime.
# 1. FA must stay OFF. --flash-attn on aborts graph build: the QSA
# indexer path calls ggml_fill on the F16-cast kq mask (ggml_fill
# asserts GGML_TYPE_F32 at ggml.c:5371). The v2 main loads and serves
# fine with --flash-attn off (verified live: model loaded, health 200,
# real completion).
# 2. The MTP draft head does NOT load on this binary. The qwen4exp loader
# requires blk.0.hc_attn_norm.weight per layer; the agentionai FAST
# head carries only blk.48 (verified 3x: EasiiX Q8_0 + FAST against
# kingjones STRIX_LEAN, FAST against this v2 main). Speculative
# decoding therefore runs ngram-mod only. The 2.28 GiB head stays on
# disk in case a compatible binary/head pair appears.
#
# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0,
# presence 1.5 — non-thinking instruct) mixed with the validated
# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096,
# t 16 / tb 16 (threads-batch 16 measured best for prefill), f16 KV, fa on,
# cont-batching, prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx
# checkpoints, --fit off (auto-fit segfaults on qwen4exp
# hyper-connection/indexer config). MTP depth 3 (n-max 3 measured best on
# this head; adaptive p-split measured worse, left off). The Qwen-Sharp
# chat template is kept on purpose (chat-template.jinja): the service keeps
# emitting reasoning in DeepSeek format for its clients.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
# incident). llama.cpp reads --host/--port/--temp as CLI flags, never as
# env vars.
@@ -37,11 +32,14 @@ LLAMA_BIN=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server
# Model package from
# https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF
# Layout: single main GGUF (per-head PLE table, fully VRAM-resident) +
# mmproj/ vision tower. The MTP draft head lives in
# /home/dank/models/mtp-Qwen3.8-Flash-Next-FAST (already on the host).
MODEL_DIR=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST
MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf
# Layout: v2/ joined-layout GGUF (PLE n-gram table as a single
# per_layer_token_embd.weight tensor — the layout the charlie12345/ROCmFPX
# qwen4exp loader requires; the root -ple16 file splits the table per head
# and only loads in the LaurentZuijdwijk Vulkan fork). mmproj/ vision tower.
# The MTP draft head lives in /home/dank/models/mtp-Qwen3.8-Flash-Next-FAST
# (already on the host).
MODEL_DIR=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2
MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf
MMPROJ_PATH=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf
# Qwen-Sharp chat template (tracked in the repo, staged on the host). The
@@ -72,9 +70,8 @@ CACHE_TYPE_V=f16
CTX_CHECKPOINTS=16
CACHE_RAM=8192
SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state
SPLIT_MODE=none
FIT=off
NGL=999
FLASH_ATTN=off
# Sampling (card instruct table, non-thinking).
TEMPERATURE=0.7
@@ -95,14 +92,16 @@ HSA_OVERRIDE_GFX_VERSION=11.5.1
VERBOSITY=4
# Speculative decoding: the model's own MTP head (agentionai
# Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF, 2.28 GiB, ROCmFP4-FAST quant
# matched to the target) at depth 3 stacked on ngram-mod. n-max 3 measured
# best (32.4 t/s, 0.612 acceptance); adaptive p-split measured worse, so it
# stays off (p-min stays at the 0.75 default). Set SPEC_TYPE=none to disable.
# Speculative decoding: ngram-mod only. The MTP draft head is retained
# (agentionai Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF, 2.28 GiB) but does
# NOT load on build-qwen38 (loader requires blk.0.hc_attn_norm.weight the
# head lacks — see KNOWN CONSTRAINTS). If a compatible binary appears, set
# SPEC_TYPE=draft-mtp,ngram-mod (n-max 3 measured best at 32.4 t/s / 0.612
# acceptance; adaptive p-split measured worse, stays off). SPEC_TYPE=none
# disables speculative decoding entirely.
DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST
DRAFT_MODEL_FILE=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf
SPEC_TYPE=draft-mtp,ngram-mod
SPEC_TYPE=ngram-mod
SPEC_DRAFT_N_MAX=3
SPEC_DRAFT_N_MIN=0
SPEC_DRAFT_P_MIN=0.75
scripts/serve-frank-large.sh
+14 -16
View File
@@ -12,18 +12,18 @@
# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
# at layer 1) quantized Q4_0_ROCMFP4_FAST, imatrix-calibrated, 87.06 GiB.
# The n-gram PLE table is split per head inside the single GGUF — fully
# VRAM-resident, no sidecar, no --ngram-on-disk. The model's own MTP
# head (2.28 GiB, matched ROCmFP4-FAST quant) drives speculative decoding
# at depth 3 (~66% draft acceptance).
# The n-gram PLE table is a single per_layer_token_embd.weight tensor —
# fully VRAM-resident, no sidecar, no --ngram-on-disk.
#
# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0,
# presence 1.5 — non-thinking instruct) mixed with the validated
# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096,
# t 16 / tb 16, f16 KV, fa on, cont-batching, prompt cache + 8 GiB
# cache-ram + idle-slot cache, 16 ctx checkpoints, --fit off (auto-fit
# segfaults on qwen4exp hyper-connection config). MTP depth 3 (n-max 3
# measured best; adaptive p-split measured worse, left off).
# t 16 / tb 16, f16 KV, fa OFF (see KNOWN CONSTRAINTS in the env file:
# flash-attn on aborts graph build on the QSA indexer path), cont-batching,
# prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx checkpoints,
# --fit off (auto-fit segfaults on qwen4exp hyper-connection config).
# Speculative decoding is ngram-mod only; the MTP head does not load on
# build-qwen38 (loader requires blk.0.hc_attn_norm.weight it lacks).
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
@@ -37,9 +37,9 @@ fi
# --- defaults (used when the env file does not set them) -------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server}"
: "${MODEL_DIR:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST}"
: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf}"
: "${MMPROJ_PATH:=$MODEL_DIR/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
: "${MODEL_DIR:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2}"
: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf}"
: "${MMPROJ_PATH:=$MODEL_DIR/../mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
: "${CHAT_TEMPLATE_KWARGS:={}}"
: "${ALIAS:=frank-default-large}"
@@ -57,17 +57,15 @@ fi
: "${CACHE_RAM:=8192}"
: "${SLOT_SAVE_PATH:=$SCRIPT_DIR/slot-state}"
: "${SPLIT_MODE:=none}"
: "${FIT:=off}"
: "${NGL:=999}"
: "${FLASH_ATTN:=off}"
: "${TEMPERATURE:=0.7}"
: "${TOP_K:=20}"
: "${TOP_P:=0.8}"
: "${MIN_P:=0.0}"
: "${PRESENCE_PENALTY:=1.5}"
: "${REASONING:=on}"
: "${REASONING_FORMAT:=deepseek}"
: "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}"
: "${SPEC_TYPE:=draft-mtp,ngram-mod}"
: "${SPEC_TYPE:=ngram-mod}"
: "${DRAFT_MODEL_DIR:=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST}"
: "${DRAFT_MODEL_FILE:=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf}"
: "${SPEC_DRAFT_N_MAX:=3}"
@@ -154,7 +152,7 @@ exec /usr/bin/env \
--batch-size "$BATCH_SIZE" \
--ubatch-size "$UBATCH_SIZE" \
--fit "$FIT" \
--flash-attn on \
--flash-attn "$FLASH_ATTN" \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
--threads "$NUM_THREADS" \
serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md
+67 -39
View File
@@ -1,7 +1,7 @@
---
type: reference
title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix, per-head PLE in-GGUF, own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large.
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large.
tags:
- llama.cpp
- rocm
@@ -17,29 +17,33 @@ timestamp: 2026-08-29
# agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
> **Active since 2026-08-29.** Replaced the jcbtc CIRU v1.1 (ciru-ai runtime)
> deployment on the same slot.
> **Active since 2026-08-29.** Replaced the kingjones STRIX_LEAN and jcbtc
> CIRU v1.1 deployments on the same slot.
> **MTP draft disabled** — the author's own head does not load on
> build-qwen38 (see Known issues #3). Speculative decoding is ngram-mod only.
`frank-default-large` (port 7770) serves
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf`) with the author's
own MTP draft head (`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout)
with the author's own MTP draft head
(`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the
charlie12345/ROCmFPX `build-qwen38` runtime.
## Why this model
- 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory
box, including the 51.2 B-parameter n-gram (PLE) table, which is split per
head inside the single GGUF (`v2/` is the joined-table layout; root is the
VRAM-resident one and is what we run).
box, including the 51.2 B-parameter n-gram (PLE) table. REPO `v2/` layout
is used (joined single `per_layer_token_embd.weight` tensor) because the
charlie12345/ROCmFPX loader requires it; the root `-ple16` file's per-head
split only loads in the LaurentZuijdwijk Vulkan fork (see Known issues #2).
- imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2
(better than AesSedai IQ4_XS / IQ3_S at smaller size).
- 18-25% faster prompt processing than a mainline quant of similar size
(356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth).
- Ships the model's own f16 vision tower (`mmproj/`) and its own MTP head
(~66% draft acceptance, mean accepted length 2.97 of 3 at n-max 3).
- Ships the model's own f16 vision tower (`mmproj/`).
- The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0
draft measured worse on this head family (acceptance + throughput).
draft measured worse on this head family — but it does not load on
build-qwen38 (Known issues #3).
## Runtime
@@ -61,7 +65,8 @@ author's own matched MTP head on this binary.
| Item | Path on Dankclaw | Size |
|---|---|---|
| Main | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) |
| Main (v2 joined) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf` | 93,484,242,816 B (87.06 GiB) |
| Main (root ple16, unused) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) |
| Vision | `…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf` | 904,004,128 B |
| Draft head | `/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf` | 2,444,519,296 B |
@@ -79,11 +84,11 @@ build-qwen38 profile:
```
llama-server \
--model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf \
--model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf \
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
--host 0.0.0.0 --port 7770 --jinja \
--device ROCm0 --gpu-layers 999 \
--split-mode none --fit off --flash-attn on \
--split-mode none --fit off --flash-attn off \
--ctx-size 262144 --parallel 1 \
--batch-size 4096 --ubatch-size 4096 \
--cache-type-k f16 --cache-type-v f16 \
@@ -94,11 +99,7 @@ llama-server \
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
--chat-template-file chat-template.jinja \
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
--spec-type draft-mtp,ngram-mod \
--spec-draft-model Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf \
--spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.75 \
--spec-draft-ngl 999 --spec-draft-device ROCm0 \
--spec-draft-type-k f16 --spec-draft-type-v f16
--spec-type ngram-mod
```
Key choices and deviations from the CIRU deployment:
@@ -107,27 +108,45 @@ Key choices and deviations from the CIRU deployment:
(same as every Flash-Next deployment).
- **`--checkpoint-min-step` removed** — build-qwen38 does not implement it
(verified via `--help`); only `--ctx-checkpoints` is passed.
- **No PLE sidecar args** — the n-gram table is in-GGUF (per-head split),
not a `ple/` CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the
- **No PLE sidecar args** — the n-gram table is in-GGUF (joined
`per_layer_token_embd.weight` from the `v2/` layout), not a `ple/`
CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the
`GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
runtime-only and are gone.
- **`--mmproj` active** — vision tower included (f16, lossless).
- **MTP n-max 3, no p-split** — author measurements: n-max 3 = 32.4 t/s /
0.612 acceptance, adaptive p-split measured worse (28.4 t/s / 0.468).
`p-min` stays at the 0.75 default.
- **`--flash-attn off` is mandatory.** With fa on, the QSA indexer path
(layers with attention compression) calls `ggml_fill` on the F16-cast kq
mask and aborts graph build (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
ggml.c:5371, backtrace through `build_attn_qsa`). Verified live on this
v2 main — loads and serves fine with fa off.
- Context kept at 262144 × 1 slot (same as the CIRU deployment; the author's
quick-start is 32768 but the full window is fine on SVM).
- MTP is disabled (ngram-mod only) — see the MTP section below.
## MTP draft head
## MTP draft head — INCOMPATIBLE on build-qwen38
The draft is agentionai's own head for this exact model:
`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, architecture
`qwen4exp` with `blk.48.hc_attn_norm.weight` present (count 1, same shape as
the CIRU draft that works). The earlier 2026-08-28 failure of this head on
the *kingjones* STRIX_LEAN main model (`missing tensor blk.0.hc_attn_norm`)
was a cross-vendor pairing problem, not a defect of the head itself. The
draft carries a 512-expert MoE, so adaptive/long drafting costs real forward
passes — keep n-max 3.
The author's own matched head (`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-
FAST-GGUF`, 2.28 GiB, `qwen4exp` arch) does **not** load on this binary:
```
error loading model: missing tensor 'blk.0.hc_attn_norm.weight'
```
The build-qwen38 qwen4exp loader requires a full per-layer
`blk.<il>.hc_attn_norm.weight` set in the draft (the graph reads all 48
layers); the head only carries `blk.48.hc_attn_norm.weight` (count 1 — all
three Flash-Next-era heads examined on this fork have that same partial
layout). Verified three times, not a cross-vendor artifact:
1. EasiiX Q8_0 head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
2. agentionai FAST head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
3. agentionai FAST head vs agentionai v2 main (2026-08-29, this deployment)
— failed identically.
The 2.28 GiB head stays on disk; the moment a binary or head appears that
carries the full per-layer norm set, re-enable with
`SPEC_TYPE=draft-mtp,ngram-mod`, `--spec-draft-n-max 3` (author-measured
best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
## Config files
@@ -152,12 +171,21 @@ loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.
1. **ROCmFP4 types require the ROCmFPX fork.** Stock llama.cpp and the CIRU
runtime reject these files.
2. **Single 87 GiB file.** Use resumable curl; allow ~10-20 minutes on a
good link.
3. **MTP head must be agentionai's.** EasiiX Q8_0 and other heads are
incompatible with this binary's draft expectations for Flash-Next
(hyper-connection norm tensor).
4. **`SVM mapping failed` kernel spam** is a documented symptom, not root
2. **Use the `v2/` joined layout, NOT the root `-ple16` split.** This binary's
qwen4exp loader (qwen4exp.cpp:104-108) unconditionally requires a single
`per_layer_token_embd.weight` tensor and GGML_ASSERTs "qwen4exp is missing
the PLE n-gram table" against the per-head split. The root file only loads
in the LaurentZuijdwijk Vulkan fork (`vulkan-qwen4exp-rocmfpx`). The v2
file is byte-identical in weights, ~5 KB larger header, loads fully into
SVM unified memory (no `--ngram-on-disk` needed — kingjones STRIX ran a
50 GiB joined PLE table the same way).
3. **No MTP draft head loads on build-qwen38.** The qwen4exp loader
requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; every
Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries
only `blk.48`. Spec decoding is ngram-mod only. See the MTP section.
4. **`--flash-attn off` is mandatory.** fa on aborts graph build on the QSA
indexer path (F32 `ggml_fill` on the F16 mask) — see Serving profile.
5. **`SVM mapping failed` kernel spam** is a documented symptom, not root
cause — investigate before blaming memory pressure.
## Citations