serverAppConfigs/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw	agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large.	llama.cpp rocm dankclaw gfx1151 agentionai rocmfp4 strix-halo qwen4exp mtp	2026-08-29

agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw

RETIRED 2026-08-30 — replaced by the kingjones STRIX model on the card's pinned kingjones30/ROCmFPX runtime; see kingjones-strix-rocmfp4 for the active runbook. This page stays as the deployment record: the v2-joined-PLE layout requirement, the MTP head incompatibility (verified 3x), and the QSA F32-mask patch (scripts/patches/qwen4exp-qsa-f32-mask.patch) that made --flash-attn on work on the charlie build-qwen38 tree — all still true for that binary + model pair, which remains the rollback target.

frank-default-large (port 7770) served agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (main, Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf, the v2/ joined layout) with the author's own MTP draft head (Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf, never loaded) on the charlie12345/ROCmFPX build-qwen38 runtime, 2026-08-29 → 2026-08-30.

Why this model

• 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory box, including the 51.2 B-parameter n-gram (PLE) table. REPO v2/ layout is used (joined single per_layer_token_embd.weight tensor) because the charlie12345/ROCmFPX loader requires it; the root -ple16 file's per-head split only loads in the LaurentZuijdwijk Vulkan fork (see Known issues #2).
• imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2 (better than AesSedai IQ4_XS / IQ3_S at smaller size).
• 18-25% faster prompt processing than a mainline quant of similar size (356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth).
• Ships the model's own f16 vision tower (mmproj/).
• The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0 draft measured worse on this head family — but it does not load on build-qwen38 (Known issues #3).

Runtime

charlie12345/ROCmFPX llama-server built at /home/dank/GitPackages/ROCmFPX/build-qwen38 (HIP, ROCm 7.2.4, gfx1151, PR #27742 qwen4exp merged). Upstream llama.cpp cannot load the ROCmFP4 quant types (TYPE_100/TYPE_101/FAST variants) — verified:

strings build-qwen38/bin/llama-server | grep Q4_0_ROCMFP4_FAST

CIRU binaries (ciru-ai v1.1) do not carry the FAST types; do not point this model at the CIRU runtime. The build previously served the kingjones STRIX / STRIX_LEAN models; the agentionai-FAST pair is the first use of the author's own matched MTP head on this binary.

Files / download

Item	Path on Dankclaw	Size
Main (v2 joined)	/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf	93,484,242,816 B (87.06 GiB)
Main (root ple16, unused)	/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf	93,484,237,760 B (87.06 GiB)
Vision	…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf	904,004,128 B
Draft head	/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf	2,444,519,296 B

Draft head sha256 (verified against HF LFS): 5046d69571bb35c699e19c7a00b36633c2fdfdd06d70482551744ec523a6b590

Downloaded with a resumable plain curl -L -C - (Hugging Face resolve/main URLs; no auth token needed for public repos). The mmproj lives in an mmproj/ subdirectory of the repo.

Serving profile

Card sampling table (instruct, non-thinking) mixed with the validated build-qwen38 profile:

llama-server \
  --model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf \
  --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
  --host 0.0.0.0 --port 7770 --jinja \
  --device ROCm0 --gpu-layers 999 \
  --split-mode none --fit off --flash-attn on \
  --ctx-size 32768 --parallel 1 \
  --batch-size 4096 --ubatch-size 4096 \
  --cache-type-k f16 --cache-type-v f16 \
  --threads 16 --threads-batch 16 \
  --ctx-checkpoints 16 --cache-ram 8192 \
  --cache-prompt --cache-idle-slots --metrics \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
  --chat-template-file chat-template.jinja \
  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
  --spec-type ngram-mod

Key choices and deviations from the CIRU deployment:

• --fit off — auto-fit segfaults on qwen4exp hyper-connection config (same as every Flash-Next deployment).
• --checkpoint-min-step removed — build-qwen38 does not implement it (verified via --help); only --ctx-checkpoints is passed.
• No PLE sidecar args — the n-gram table is in-GGUF (joined per_layer_token_embd.weight from the v2/ layout), not a ple/ CIRUPLE1 sidecar. --ple-sidecar/--ple-cache-mib and the GGML_QWEN4EXP_PLE_* / GGML_CUDA_Q41_MOE_FORCE_J env knobs are CIRU- runtime-only and are gone.
• --mmproj active — vision tower included (f16, lossless).
• --flash-attn on (restored 2026-08-29). fa on previously aborted graph build: the QSA indexer path (layers with attention compression) called ggml_fill on the F16-cast kq mask (GGML_ASSERT(a->type == GGML_TYPE_F32), ggml.c:5371, backtrace through build_attn_qsa). Fixed in the fork source by the QSA F32-mask patch — vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch (sha256 f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f); the binary at build-qwen38/bin/llama-server (rebuilt 2026-08-29) already carries it. A clean rebuild must re-apply git apply scripts/patches/qwen4exp-qsa-f32-mask.patch inside /home/dank/GitPackages/ROCmFPX before make. Verified live at CTX_SIZE=32768: model loads, health 200, real completion.
• Context capped at 32768 × 1 slot — the author's own quick-start. This model is fully GPU-resident (87.06 GiB): the PLE n-gram table sits inside the GGUF as one tensor, unlike STRIX_LEAN which streamed it from RAM. HIP heap is 119.3 GiB; the context buffer is 50.8 GiB at 131072 and 103 GiB at 262144, so either window OOMs (cudaMalloc … failed: out of memory, verified live on first graph alloc). At 32768 the buffer is ~13 GiB (~100 GiB total) and loads clean.
• MTP is disabled (ngram-mod only) — see the MTP section below.

MTP draft head — INCOMPATIBLE on build-qwen38

The author's own matched head (agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4- FAST-GGUF, 2.28 GiB, qwen4exp arch) does not load on this binary:

error loading model: missing tensor 'blk.0.hc_attn_norm.weight'

The build-qwen38 qwen4exp loader requires a full per-layer blk.<il>.hc_attn_norm.weight set in the draft (the graph reads all 48 layers); the head only carries blk.48.hc_attn_norm.weight (count 1 — all three Flash-Next-era heads examined on this fork have that same partial layout). Verified three times, not a cross-vendor artifact:

1. EasiiX Q8_0 head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
2. agentionai FAST head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
3. agentionai FAST head vs agentionai v2 main (2026-08-29, this deployment) — failed identically.

The 2.28 GiB head stays on disk; the moment a binary or head appears that carries the full per-layer norm set, re-enable with SPEC_TYPE=draft-mtp,ngram-mod, --spec-draft-n-max 3 (author-measured best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).

Config files

• Env: scripts/frank-default-large.env
• Serve: scripts/serve-frank-large.sh
• Chat template: scripts/chat-template.jinja (unchanged, Qwen-Sharp)
• Fork patch: scripts/patches/qwen4exp-qsa-f32-mask.patch (re-apply after any clean fork rebuild — see Serving profile)

Swap procedure

sudo systemctl stop frank-default-large
# edit scripts/frank-default-large.env + scripts/serve-frank-large.sh, commit, push
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-default-large
sudo systemctl start frank-default-large   # or via deploy.sh
curl http://192.168.0.143:7770/health

Model load is slow (87 GiB + 2.3 GiB draft): health returns 503 while loading; deploy.sh waits up to 20 minutes (LLM_WAIT). Do not kill it.

Known issues

1. ROCmFP4 types require the ROCmFPX fork. Stock llama.cpp and the CIRU runtime reject these files.
2. Use the v2/ joined layout, NOT the root -ple16 split. This binary's qwen4exp loader (qwen4exp.cpp:104-108) unconditionally requires a single per_layer_token_embd.weight tensor and GGML_ASSERTs "qwen4exp is missing the PLE n-gram table" against the per-head split. The root file only loads in the LaurentZuijdwijk Vulkan fork (vulkan-qwen4exp-rocmfpx). The v2 file is byte-identical in weights, ~5 KB larger header, loads fully into SVM unified memory (no --ngram-on-disk needed — kingjones STRIX ran a 50 GiB joined PLE table the same way).
3. No MTP draft head loads on build-qwen38. The qwen4exp loader requires a full per-layer blk.<il>.hc_attn_norm.weight set; every Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries only blk.48. Spec decoding is ngram-mod only. See the MTP section.
4. --flash-attn on (restored 2026-08-29). fa on aborts graph build without the QSA F32-mask patch — see Serving profile (fix + sha256 vendored there).
5. SVM mapping failed kernel spam is a documented symptom, not root cause — investigate before blaming memory pressure.

Citations

• agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF
• agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF
• Qwen/Qwen3.8-Flash-Next base model
• charlie12345/ROCmFPX — fork with PR #27742 (qwen4exp architecture) and the ROCmFP4 types
• llama.cpp PR #27742 — qwen4exp support (merged upstream)
• llama.cpp PR #27739 — MTP draft head support