serverAppConfigs/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw	agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix, per-head PLE in-GGUF, own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large.	llama.cpp rocm dankclaw gfx1151 agentionai rocmfp4 strix-halo qwen4exp mtp	2026-08-29

agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw

Active since 2026-08-29. Replaced the jcbtc CIRU v1.1 (ciru-ai runtime) deployment on the same slot.

frank-default-large (port 7770) serves agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (main, Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf) with the author's own MTP draft head (Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf) on the charlie12345/ROCmFPX build-qwen38 runtime.

Why this model

• 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory box, including the 51.2 B-parameter n-gram (PLE) table, which is split per head inside the single GGUF (v2/ is the joined-table layout; root is the VRAM-resident one and is what we run).
• imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2 (better than AesSedai IQ4_XS / IQ3_S at smaller size).
• 18-25% faster prompt processing than a mainline quant of similar size (356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth).
• Ships the model's own f16 vision tower (mmproj/) and its own MTP head (~66% draft acceptance, mean accepted length 2.97 of 3 at n-max 3).
• The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0 draft measured worse on this head family (acceptance + throughput).

Runtime

charlie12345/ROCmFPX llama-server built at /home/dank/GitPackages/ROCmFPX/build-qwen38 (HIP, ROCm 7.2.4, gfx1151, PR #27742 qwen4exp merged). Upstream llama.cpp cannot load the ROCmFP4 quant types (TYPE_100/TYPE_101/FAST variants) — verified:

strings build-qwen38/bin/llama-server | grep Q4_0_ROCMFP4_FAST

CIRU binaries (ciru-ai v1.1) do not carry the FAST types; do not point this model at the CIRU runtime. The build previously served the kingjones STRIX / STRIX_LEAN models; the agentionai-FAST pair is the first use of the author's own matched MTP head on this binary.

Files / download

Item	Path on Dankclaw	Size
Main	/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf	93,484,237,760 B (87.06 GiB)
Vision	…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf	904,004,128 B
Draft head	/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf	2,444,519,296 B

Draft head sha256 (verified against HF LFS): 5046d69571bb35c699e19c7a00b36633c2fdfdd06d70482551744ec523a6b590

Downloaded with a resumable plain curl -L -C - (Hugging Face resolve/main URLs; no auth token needed for public repos). The mmproj lives in an mmproj/ subdirectory of the repo.

Serving profile

Card sampling table (instruct, non-thinking) mixed with the validated build-qwen38 profile:

llama-server \
  --model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf \
  --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
  --host 0.0.0.0 --port 7770 --jinja \
  --device ROCm0 --gpu-layers 999 \
  --split-mode none --fit off --flash-attn on \
  --ctx-size 262144 --parallel 1 \
  --batch-size 4096 --ubatch-size 4096 \
  --cache-type-k f16 --cache-type-v f16 \
  --threads 16 --threads-batch 16 \
  --ctx-checkpoints 16 --cache-ram 8192 \
  --cache-prompt --cache-idle-slots --metrics \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
  --chat-template-file chat-template.jinja \
  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-model Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf \
  --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.75 \
  --spec-draft-ngl 999 --spec-draft-device ROCm0 \
  --spec-draft-type-k f16 --spec-draft-type-v f16

Key choices and deviations from the CIRU deployment:

• --fit off — auto-fit segfaults on qwen4exp hyper-connection config (same as every Flash-Next deployment).
• --checkpoint-min-step removed — build-qwen38 does not implement it (verified via --help); only --ctx-checkpoints is passed.
• No PLE sidecar args — the n-gram table is in-GGUF (per-head split), not a ple/ CIRUPLE1 sidecar. --ple-sidecar/--ple-cache-mib and the GGML_QWEN4EXP_PLE_* / GGML_CUDA_Q41_MOE_FORCE_J env knobs are CIRU- runtime-only and are gone.
• --mmproj active — vision tower included (f16, lossless).
• MTP n-max 3, no p-split — author measurements: n-max 3 = 32.4 t/s / 0.612 acceptance, adaptive p-split measured worse (28.4 t/s / 0.468). p-min stays at the 0.75 default.
• Context kept at 262144 × 1 slot (same as the CIRU deployment; the author's quick-start is 32768 but the full window is fine on SVM).

MTP draft head

The draft is agentionai's own head for this exact model: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF, 2.28 GiB, architecture qwen4exp with blk.48.hc_attn_norm.weight present (count 1, same shape as the CIRU draft that works). The earlier 2026-08-28 failure of this head on the kingjones STRIX_LEAN main model (missing tensor blk.0.hc_attn_norm) was a cross-vendor pairing problem, not a defect of the head itself. The draft carries a 512-expert MoE, so adaptive/long drafting costs real forward passes — keep n-max 3.

Config files

• Env: scripts/frank-default-large.env
• Serve: scripts/serve-frank-large.sh
• Chat template: scripts/chat-template.jinja (unchanged, Qwen-Sharp)

Swap procedure

sudo systemctl stop frank-default-large
# edit scripts/frank-default-large.env + scripts/serve-frank-large.sh, commit, push
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-default-large
sudo systemctl start frank-default-large   # or via deploy.sh
curl http://192.168.0.143:7770/health

Model load is slow (87 GiB + 2.3 GiB draft): health returns 503 while loading; deploy.sh waits up to 20 minutes (LLM_WAIT). Do not kill it.

Known issues

1. ROCmFP4 types require the ROCmFPX fork. Stock llama.cpp and the CIRU runtime reject these files.
2. Single 87 GiB file. Use resumable curl; allow ~10-20 minutes on a good link.
3. MTP head must be agentionai's. EasiiX Q8_0 and other heads are incompatible with this binary's draft expectations for Flash-Next (hyper-connection norm tensor).
4. SVM mapping failed kernel spam is a documented symptom, not root cause — investigate before blaming memory pressure.

Citations

• agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF
• agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF
• Qwen/Qwen3.8-Flash-Next base model
• charlie12345/ROCmFPX — fork with PR #27742 (qwen4exp architecture) and the ROCmFP4 types
• llama.cpp PR #27742 — qwen4exp support (merged upstream)
• llama.cpp PR #27739 — MTP draft head support