serverAppConfigs/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md

---
type: reference
title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix, per-head PLE in-GGUF, own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large.
tags:
  - llama.cpp
  - rocm
  - dankclaw
  - gfx1151
  - agentionai
  - rocmfp4
  - strix-halo
  - qwen4exp
  - mtp
timestamp: 2026-08-29
---

# agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw

> **Active since 2026-08-29.** Replaced the jcbtc CIRU v1.1 (ciru-ai runtime)
> deployment on the same slot.

`frank-default-large` (port 7770) serves
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf`) with the author's
own MTP draft head (`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the
charlie12345/ROCmFPX `build-qwen38` runtime.

## Why this model

- 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory
  box, including the 51.2 B-parameter n-gram (PLE) table, which is split per
  head inside the single GGUF (`v2/` is the joined-table layout; root is the
  VRAM-resident one and is what we run).
- imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2
  (better than AesSedai IQ4_XS / IQ3_S at smaller size).
- 18-25% faster prompt processing than a mainline quant of similar size
  (356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth).
- Ships the model's own f16 vision tower (`mmproj/`) and its own MTP head
  (~66% draft acceptance, mean accepted length 2.97 of 3 at n-max 3).
- The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0
  draft measured worse on this head family (acceptance + throughput).

## Runtime

charlie12345/ROCmFPX `llama-server` built at
`/home/dank/GitPackages/ROCmFPX/build-qwen38` (HIP, ROCm 7.2.4, gfx1151,
PR #27742 qwen4exp merged). Upstream llama.cpp cannot load the ROCmFP4
quant types (`TYPE_100`/`TYPE_101`/FAST variants) — verified:

```
strings build-qwen38/bin/llama-server | grep Q4_0_ROCMFP4_FAST
```

CIRU binaries (ciru-ai v1.1) do **not** carry the FAST types; do not point
this model at the CIRU runtime. The build previously served the kingjones
STRIX / STRIX_LEAN models; the agentionai-FAST pair is the first use of the
author's own matched MTP head on this binary.

## Files / download

| Item | Path on Dankclaw | Size |
|---|---|---|
| Main | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) |
| Vision | `…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf` | 904,004,128 B |
| Draft head | `/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf` | 2,444,519,296 B |

Draft head sha256 (verified against HF LFS):
`5046d69571bb35c699e19c7a00b36633c2fdfdd06d70482551744ec523a6b590`

Downloaded with a resumable plain `curl -L -C -` (Hugging Face `resolve/main`
URLs; no auth token needed for public repos). The mmproj lives in an `mmproj/`
subdirectory of the repo.

## Serving profile

Card sampling table (instruct, non-thinking) mixed with the validated
build-qwen38 profile:

```
llama-server \
  --model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf \
  --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
  --host 0.0.0.0 --port 7770 --jinja \
  --device ROCm0 --gpu-layers 999 \
  --split-mode none --fit off --flash-attn on \
  --ctx-size 262144 --parallel 1 \
  --batch-size 4096 --ubatch-size 4096 \
  --cache-type-k f16 --cache-type-v f16 \
  --threads 16 --threads-batch 16 \
  --ctx-checkpoints 16 --cache-ram 8192 \
  --cache-prompt --cache-idle-slots --metrics \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
  --chat-template-file chat-template.jinja \
  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-model Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf \
  --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.75 \
  --spec-draft-ngl 999 --spec-draft-device ROCm0 \
  --spec-draft-type-k f16 --spec-draft-type-v f16
```

Key choices and deviations from the CIRU deployment:

- **`--fit off`** — auto-fit segfaults on qwen4exp hyper-connection config
  (same as every Flash-Next deployment).
- **`--checkpoint-min-step` removed** — build-qwen38 does not implement it
  (verified via `--help`); only `--ctx-checkpoints` is passed.
- **No PLE sidecar args** — the n-gram table is in-GGUF (per-head split),
  not a `ple/` CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the
  `GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
  runtime-only and are gone.
- **`--mmproj` active** — vision tower included (f16, lossless).
- **MTP n-max 3, no p-split** — author measurements: n-max 3 = 32.4 t/s /
  0.612 acceptance, adaptive p-split measured worse (28.4 t/s / 0.468).
  `p-min` stays at the 0.75 default.
- Context kept at 262144 × 1 slot (same as the CIRU deployment; the author's
  quick-start is 32768 but the full window is fine on SVM).

## MTP draft head

The draft is agentionai's own head for this exact model:
`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, architecture
`qwen4exp` with `blk.48.hc_attn_norm.weight` present (count 1, same shape as
the CIRU draft that works). The earlier 2026-08-28 failure of this head on
the *kingjones* STRIX_LEAN main model (`missing tensor blk.0.hc_attn_norm`)
was a cross-vendor pairing problem, not a defect of the head itself. The
draft carries a 512-expert MoE, so adaptive/long drafting costs real forward
passes — keep n-max 3.

## Config files

- Env: `scripts/frank-default-large.env`
- Serve: `scripts/serve-frank-large.sh`
- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)

## Swap procedure

```bash
sudo systemctl stop frank-default-large
# edit scripts/frank-default-large.env + scripts/serve-frank-large.sh, commit, push
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-default-large
sudo systemctl start frank-default-large   # or via deploy.sh
curl http://192.168.0.143:7770/health
```

Model load is slow (87 GiB + 2.3 GiB draft): health returns 503 while
loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.

## Known issues

1. **ROCmFP4 types require the ROCmFPX fork.** Stock llama.cpp and the CIRU
   runtime reject these files.
2. **Single 87 GiB file.** Use resumable curl; allow ~10-20 minutes on a
   good link.
3. **MTP head must be agentionai's.** EasiiX Q8_0 and other heads are
   incompatible with this binary's draft expectations for Flash-Next
   (hyper-connection norm tensor).
4. **`SVM mapping failed` kernel spam** is a documented symptom, not root
   cause — investigate before blaming memory pressure.

## Citations

- [agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF)
- [agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF)
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture) and the ROCmFP4 types
- [llama.cpp PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742) — qwen4exp support (merged upstream)
- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support