frank-default-large: swap to agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix + own MTP draft

Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB, Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF — fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST (2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3). Vision tower (f16) loaded via mmproj/. Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) — the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1 deployment (ciru-ai runtime). Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0, presence 1.5) mixed with the validated build-qwen38 serving profile: 262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on, cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx checkpoints (no --checkpoint-min-step — unsupported on this binary), --fit off (auto-fit segfaults on qwen4exp hyper-connection config), --spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured worse per author card), p-min 0.75. Qwen-Sharp chat template kept; reasoning stays in DeepSeek format. The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN model was a cross-vendor pairing problem: the head carries blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
2026-08-29 20:41:12 -07:00
parent e22631d892
commit 3c2a5d2fe7
6 changed files with 313 additions and 140 deletions
@@ -27,7 +27,7 @@ host.
 | Target | Runtime | Service | Model | Port |
 |--------|---------|---------|-------|------|
 | Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
-| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8 Uncensored Q6_0_ROCmFPX_AGENT (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 |
+| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head (ROCmFP4-FAST), 262K context / 1 slot, reasoning on | 7770 |

 History: the 2026-08-17 redo retired the pipeline-parallel split and the
 FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
@@ -274,27 +274,36 @@ When you rename a service or move a model to a new host, update

 ## Runtime & Troubleshooting

-### Qwen3.8-Flash-Next ROCmFP4 STRIX_LEAN llama.cpp (Dankclaw, ACTIVE)
+### Qwen3.8-Flash-Next ROCmFP4-FAST imatrix llama.cpp (Dankclaw, ACTIVE)

 The large service serves
-`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF` (public, 98.5 GiB)
-from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN/`
-with a charlie12345/ROCmFPX `llama-server` built at
+`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (public,
+87.06 GiB, Q4_0_ROCMFP4_FAST imatrix-calibrated) from
+`/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/` with a
+charlie12345/ROCmFPX `llama-server` built at
 `/home/dank/GitPackages/ROCmFPX/build-qwen38` (PR #27742 qwen4exp architecture
 merged). The model is a 56B MoE (512 experts, 10 used), 48 layers (12 attn +
 36 SSM, full_attention_interval=4), QSA indexer (4 heads, top_k=2048),
-hyper-connections (count=4, low_rank=320), PLE (layer 1). Q4_0_ROCMFP4 quant.
-It runs f16 KV cache, 131072 context / 1 slot, `--fit off` (auto-fit segfaults
-on qwen4exp hyper-connection config), and reasoning on (Sharp template, budget
-1, `--reasoning-format deepseek`; chat kwargs keep `preserve_thinking` +
-`preserve_reasoning`). Env `scripts/frank-default-large.env`, serve
-`scripts/serve-frank-large.sh`.
+hyper-connections (count=4, low_rank=320), PLE (layer 1). The 51.2 B-parameter
+n-gram table is split per head INSIDE the single GGUF — fully VRAM-resident,
+no sidecar, no `--ngram-on-disk`. The model's own MTP draft head
+(`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, ROCmFP4-FAST
+quant matched to target, ~66% draft acceptance) drives speculative decoding
+at depth 3; vision tower (f16) ships as mmproj/. It runs 262144 context / 1
+slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, `--fit off` (auto-fit
+segfaults on qwen4exp hyper-connection config), and reasoning on (Sharp
+template, budget -1, `--reasoning-format deepseek`; chat kwargs keep
+`preserve_thinking` + `preserve_reasoning`). Env
+`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`.

-**Serving status (2026-09-01):** deployed the Flash-Next 56B MoE model. The
-previous kingjones Q6_0 AGENT model (27B, standalone MTP, 262144 context) is
-retired. The Q8_0 AGENT experiment is documented in
-`serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md`; it was retired because it crashed in
-`ggml_compute_forward_set_rows` and did not provide a working service.
+**Serving status (2026-08-29):** deployed the agentionai ROCmFP4-FAST imatrix
+model with its own matched MTP draft head, replacing the jcbtc CIRU v1.1
+deployment (ciru-ai runtime). The 2026-08-28 failure of this head on the
+kingjones STRIX_LEAN model (`missing tensor blk.0.hc_attn_norm.weight`) was a
+cross-vendor pairing problem — the head carries the hyper-connection norm
+tensor (`blk.48.hc_attn_norm.weight`) and works against its own main file.
+Runbook:
+[serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md).

 ### Tiel-Coder-35B-A3B llama.cpp (Dankserver, ACTIVE)

@@ -1,45 +1,48 @@
-# Tunables for frank-default-large (jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 on
-# the ciru-ai Qwen3.8-Flash-CIRU-STRIX-IU4 llama.cpp runtime, tag v1.1,
-# Dankclaw, Strix Halo gfx1151 unified memory).
+# Tunables for frank-default-large (agentionai/Qwen3.8-Flash-Next-ROCmFP4-
+# FAST-imatrix-GGUF on the charlie12345/ROCmFPX llama.cpp runtime,
+# build-qwen38, Dankclaw, Strix Halo gfx1151 unified memory).
 #
-# Runtime: full llama.cpp fork at
-# /home/dank/GitPackages/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4/build-gfx1151,
-# built with the vendor build-linux-amd.sh flag set (HIP on, HIP_NO_VMM on,
-# HIP_GRAPHS on, HIP_MMQ_MFMA on, CUDA/VULKAN/SYCL off, gfx1151) plus
-# GGML_CUDA_FA + GGML_CUDA_FA_ALL_QUANTS. The IU4 WMMA expert path and the
-# CIRUPLE1 PLE pager read the GGML_* env knobs exported by the serve script.
-# Replaced the kingjones STRIX_LEAN (ROCmFPX build-qwen38) deployment.
+# Runtime: charlie12345/ROCmFPX llama.cpp at
+# /home/dank/GitPackages/ROCmFPX/build-qwen38 (HIP, ROCm 7.2.4, gfx1151,
+# PR #27742 qwen4exp merged). The ROCmFP4 quant types are hand-ported from
+# ciru-ai/ROCmFPX; upstream llama.cpp cannot load this GGUF. Replaced the
+# jcbtc CIRU v1.1 (ciru-ai runtime) and kingjones STRIX_LEAN deployments.
 #
-# Model: Qwen3.8-Flash-Next base (56B MoE, 512 experts / 10 used, 48 layers
-# = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE at layer 1)
-# in the CIRU quality-first mixed quant: 144 routed-expert tensors at Q4_1
-# on the native IU4 WMMA path, protected tensors at Q5_K/Q5_1/Q8_0/BF16/F32.
-# Main GGUF ~79 GB + MTP Q8_0 draft ~4 GB + exact-FP8 PLE sidecar ~52 GB
-# paged from disk (never in VRAM; 4 GiB decoded-page cache via
-# --ple-cache-mib). Vendor-validated on the same silicon (gfx1151).
+# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
+# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
+# at layer 1) quantized Q4_0_ROCMFP4_FAST imatrix-calibrated, 87.06 GiB,
+# 4.23 bpw. The n-gram PLE table is split per head inside the single GGUF
+# (fully VRAM-resident — no ple sidecar, no --ngram-on-disk). Vision tower
+# (f16, lossless) ships as mmproj/. The model's own MTP head (2.28 GiB,
+# matched ROCmFP4-FAST quant, ~66% draft acceptance) drives spec decoding.
 #
-# The vendor production profile is applied verbatim on this first deployment:
-# -ngl all, -sm none, --fit off, ctx 262144 x 1 slot, b 2048 / ub 512,
-# t 8 / tb 8, f16 KV, fa on, cont-batching, prompt cache + 8 GiB cache-ram
-# + idle-slot cache, 32 ctx checkpoints with min step 8192, MTP depth 3
-# with p-split 0.10, GGML_CUDA_Q41_MOE_FORCE_J=32,
-# GGML_QWEN4EXP_PLE_WORKERS=16, GGML_QWEN4EXP_PLE_STRICT_SHA=0 (checksums
-# were verified with sha256sum -c at download time), ROCBLAS_USE_HIPBLASLT=1.
-# The Qwen-Sharp chat template is kept on purpose (chat-template.jinja): the
-# service keeps emitting reasoning in DeepSeek format for its clients.
+# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0,
+# presence 1.5 — non-thinking instruct) mixed with the validated
+# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096,
+# t 16 / tb 16 (threads-batch 16 measured best for prefill), f16 KV, fa on,
+# cont-batching, prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx
+# checkpoints, --fit off (auto-fit segfaults on qwen4exp
+# hyper-connection/indexer config). MTP depth 3 (n-max 3 measured best on
+# this head; adaptive p-split measured worse, left off). The Qwen-Sharp
+# chat template is kept on purpose (chat-template.jinja): the service keeps
+# emitting reasoning in DeepSeek format for its clients.
 #
 # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
 # ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
 # incident). llama.cpp reads --host/--port/--temp as CLI flags, never as
 # env vars.

-# CIRU v1.1 runtime binary (vendor build script, plus FA_ALL_QUANTS).
-LLAMA_BIN=/home/dank/GitPackages/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4/build-gfx1151/bin/llama-server
+# ROCmFPX (build-qwen38) runtime binary.
+LLAMA_BIN=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server

-# Model package from https://huggingface.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4
-# Layout: main GGUF + mtp/ Q8_0 draft + ple/ CIRUPLE1 sidecar (mandatory).
-MODEL_DIR=/home/dank/models/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4
-MODEL_FILE=Qwen3.8-Flash-CIRU-STRIX-IU4.gguf
+# Model package from
+# https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF
+# Layout: single main GGUF (per-head PLE table, fully VRAM-resident) +
+# mmproj/ vision tower. The MTP draft head lives in
+# /home/dank/models/mtp-Qwen3.8-Flash-Next-FAST (already on the host).
+MODEL_DIR=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST
+MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf
+MMPROJ_PATH=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf

 # Qwen-Sharp chat template (tracked in the repo, staged on the host). The
 # Sharp template drives thinking on by default (medium effort); the server
@@ -53,28 +56,27 @@ ALIAS=frank-default-large
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7770

-# Vendor production profile, verbatim: 262144-token context on ONE slot
-# (parallel 1), batch 2048 / ubatch 512, 8+8 threads, f16 KV, 32 context
-# checkpoints with min step 8192, 8 GiB prompt cache, 4 GiB PLE page cache.
-# --fit off: auto-fit segfaults on this model's hyper-connection/indexer
-# config; -ngl all puts every layer on the SVM-backed GPU pool.
+# Serving: 262144-token context on ONE slot (parallel 1), batch 4096 /
+# ubatch 4096, 16+16 threads, f16 KV, 16 context checkpoints, 8 GiB
+# prompt cache. --fit off: auto-fit segfaults on this model's
+# hyper-connection/indexer config; -ngl 999 puts every layer on the
+# SVM-backed GPU pool.
 CTX_SIZE=262144
 PARALLEL=1
-BATCH_SIZE=2048
-UBATCH_SIZE=512
-NUM_THREADS=8
-THREAD_BATCH=8
+BATCH_SIZE=4096
+UBATCH_SIZE=4096
+NUM_THREADS=16
+THREAD_BATCH=16
 CACHE_TYPE_K=f16
 CACHE_TYPE_V=f16
-CTX_CHECKPOINTS=32
-CHECKPOINT_MIN_STEP=8192
+CTX_CHECKPOINTS=16
 CACHE_RAM=8192
-PLE_CACHE_MIB=4096
 SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state
 SPLIT_MODE=none
 FIT=off
+NGL=999

-# Sampling (Sharp default; vendor API examples agree for non-thinking mode).
+# Sampling (card instruct table, non-thinking).
 TEMPERATURE=0.7
 TOP_K=20
 TOP_P=0.8
@@ -93,26 +95,18 @@ HSA_OVERRIDE_GFX_VERSION=11.5.1

 VERBOSITY=4

-# Speculative decoding: MTP draft head (Q8_0) at depth 3, p-split 0.10 —
-# the vendor headline profile. ngram-mod stacking is a later experiment;
-# SPEC_TYPE=none disables spec decoding entirely.
-DRAFT_MODEL_DIR=/home/dank/models/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4/mtp
-DRAFT_MODEL_FILE=Qwen3.8-Flash-CIRU-STRIX-IU4-MTP-Q8_0.gguf
-SPEC_TYPE=draft-mtp
+# Speculative decoding: the model's own MTP head (agentionai
+# Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF, 2.28 GiB, ROCmFP4-FAST quant
+# matched to the target) at depth 3 stacked on ngram-mod. n-max 3 measured
+# best (32.4 t/s, 0.612 acceptance); adaptive p-split measured worse, so it
+# stays off (p-min stays at the 0.75 default). Set SPEC_TYPE=none to disable.
+DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST
+DRAFT_MODEL_FILE=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf
+SPEC_TYPE=draft-mtp,ngram-mod
 SPEC_DRAFT_N_MAX=3
 SPEC_DRAFT_N_MIN=0
-SPEC_DRAFT_P_MIN=0.0
-SPEC_DRAFT_P_SPLIT=0.10
-SPEC_DRAFT_NGL=all
+SPEC_DRAFT_P_MIN=0.75
+SPEC_DRAFT_NGL=999
 SPEC_DRAFT_DEVICE=ROCm0
-SPEC_DRAFT_TYPE_K=q8_0
-SPEC_DRAFT_TYPE_V=q8_0
-
-# Vendor runtime env knobs (read by the hipified mmq + PLE pager kernels).
-# GGML_CUDA_Q41_MOE_FORCE_J forces the Q4_1 MoE WMMA tile width; the PLE
-# pager knobs control the CIRUPLE1 page-in workers and skip re-hashing the
-# 52 GB payload at every launch (checksums were verified once, at download).
-GGML_CUDA_Q41_MOE_FORCE_J=32
-GGML_QWEN4EXP_PLE_WORKERS=16
-GGML_QWEN4EXP_PLE_STRICT_SHA=0
-ROCBLAS_USE_HIPBLASLT=1
+SPEC_DRAFT_TYPE_K=f16
+SPEC_DRAFT_TYPE_V=f16
@@ -1,28 +1,29 @@
 #!/usr/bin/env bash
-# Serve jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 as frank-default-large via the
-# ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4 llama.cpp runtime (tag v1.1) on
-# Dankclaw (Strix Halo gfx1151, unified memory).
+# Serve agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF as
+# frank-default-large via the charlie12345/ROCmFPX llama.cpp runtime
+# (build-qwen38, PR #27742 qwen4exp merged) on Dankclaw (Strix Halo
+# gfx1151, unified memory).
 #
-# Runtime: ciru-ai Qwen3.8-Flash-CIRU-STRIX-IU4 llama.cpp built at
-# /home/dank/GitPackages/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4/build-gfx1151
-# with the vendor build-linux-amd.sh flag set (HIP on, HIP_NO_VMM on,
-# HIP_GRAPHS on, HIP_MMQ_MFMA on, CUDA/VULKAN/SYCL off, gfx1151) PLUS
-# -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON. The Q4_1_MOE_FORCE_J and
-# PLE env knobs below are read by the hipified mmq/pager kernels.
+# Runtime: charlie12345/ROCmFPX llama.cpp built at
+# /home/dank/GitPackages/ROCmFPX/build-qwen38 (HIP, gfx1151, ROCm 7.2.4).
+# The ROCmFP4 quant types (Q4_0_ROCMFP4_FAST) are hand-ported from
+# ciru-ai/ROCmFPX; upstream llama.cpp cannot load this GGUF.
 #
-# Model: Qwen3.8-Flash-Next base (56B MoE, 512 experts / 10 used, 48 layers
-# = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE at layer 1)
-# in the CIRU quality-first mixed quant: 144 routed-expert tensors at Q4_1
-# on the native IU4 WMMA path, protected tensors at Q5_K/Q5_1/Q8_0/BF16/F32.
-# The exact-FP8 PLE table is a sidecar (ple/ dir) paged from disk through the
-# CIRUPLE1 pager with a ple-cache-mib decoded-page cache; it never enters
-# VRAM.  The MTP draft head (Q8_0) drives speculative decoding at depth 3.
+# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
+# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
+# at layer 1) quantized Q4_0_ROCMFP4_FAST, imatrix-calibrated, 87.06 GiB.
+# The n-gram PLE table is split per head inside the single GGUF — fully
+# VRAM-resident, no sidecar, no --ngram-on-disk.  The model's own MTP
+# head (2.28 GiB, matched ROCmFP4-FAST quant) drives speculative decoding
+# at depth 3 (~66% draft acceptance).
 #
-# Vendor production profile is applied verbatim on the first deployment:
-# -ngl all, -sm none, --fit off, ctx 262144 x 1 slot, b 2048 / ub 512,
-# t 8 / tb 8, f16 KV, fa on, cont-batching, prompt cache + 8 GiB cache-ram
-# + idle-slot cache, 32 ctx checkpoints with min step 8192, MTP depth 3
-# with p-split 0.10.  Deviations from the vendor launcher are marked below.
+# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0,
+# presence 1.5 — non-thinking instruct) mixed with the validated
+# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096,
+# t 16 / tb 16, f16 KV, fa on, cont-batching, prompt cache + 8 GiB
+# cache-ram + idle-slot cache, 16 ctx checkpoints, --fit off (auto-fit
+# segfaults on qwen4exp hyper-connection config).  MTP depth 3 (n-max 3
+# measured best; adaptive p-split measured worse, left off).
 set -euo pipefail

 SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
@@ -35,9 +36,10 @@ if [[ -f "$ENV_FILE" ]]; then
 fi

 # --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4/build-gfx1151/bin/llama-server}"
-: "${MODEL_DIR:=/home/dank/models/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4}"
-: "${MODEL_FILE:=Qwen3.8-Flash-CIRU-STRIX-IU4.gguf}"
+: "${LLAMA_BIN:=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server}"
+: "${MODEL_DIR:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST}"
+: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf}"
+: "${MMPROJ_PATH:=$MODEL_DIR/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
 : "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
 : "${CHAT_TEMPLATE_KWARGS:={}}"
 : "${ALIAS:=frank-default-large}"
@@ -45,19 +47,18 @@ fi
 : "${LISTEN_PORT:=7770}"
 : "${CTX_SIZE:=262144}"
 : "${PARALLEL:=1}"
-: "${BATCH_SIZE:=2048}"
-: "${UBATCH_SIZE:=512}"
-: "${NUM_THREADS:=8}"
-: "${THREAD_BATCH:=8}"
+: "${BATCH_SIZE:=4096}"
+: "${UBATCH_SIZE:=4096}"
+: "${NUM_THREADS:=16}"
+: "${THREAD_BATCH:=16}"
 : "${CACHE_TYPE_K:=f16}"
 : "${CACHE_TYPE_V:=f16}"
-: "${CTX_CHECKPOINTS:=32}"
-: "${CHECKPOINT_MIN_STEP:=8192}"
+: "${CTX_CHECKPOINTS:=16}"
 : "${CACHE_RAM:=8192}"
-: "${PLE_CACHE_MIB:=4096}"
 : "${SLOT_SAVE_PATH:=$SCRIPT_DIR/slot-state}"
 : "${SPLIT_MODE:=none}"
 : "${FIT:=off}"
+: "${NGL:=999}"
 : "${TEMPERATURE:=0.7}"
 : "${TOP_K:=20}"
 : "${TOP_P:=0.8}"
@@ -66,20 +67,19 @@ fi
 : "${REASONING:=on}"
 : "${REASONING_FORMAT:=deepseek}"
 : "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}"
-: "${SPEC_TYPE:=draft-mtp}"
-: "${DRAFT_MODEL_DIR:=$MODEL_DIR/mtp}"
-: "${DRAFT_MODEL_FILE:=Qwen3.8-Flash-CIRU-STRIX-IU4-MTP-Q8_0.gguf}"
+: "${SPEC_TYPE:=draft-mtp,ngram-mod}"
+: "${DRAFT_MODEL_DIR:=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST}"
+: "${DRAFT_MODEL_FILE:=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf}"
 : "${SPEC_DRAFT_N_MAX:=3}"
 : "${SPEC_DRAFT_N_MIN:=0}"
-: "${SPEC_DRAFT_P_MIN:=0.0}"
-: "${SPEC_DRAFT_P_SPLIT:=0.10}"
-: "${SPEC_DRAFT_NGL:=all}"
+: "${SPEC_DRAFT_P_MIN:=0.75}"
+: "${SPEC_DRAFT_NGL:=999}"
 : "${SPEC_DRAFT_DEVICE:=ROCm0}"
-: "${SPEC_DRAFT_TYPE_K:=q8_0}"
-: "${SPEC_DRAFT_TYPE_V:=q8_0}"
+: "${SPEC_DRAFT_TYPE_K:=f16}"
+: "${SPEC_DRAFT_TYPE_V:=f16}"

 MODEL="$MODEL_DIR/$MODEL_FILE"
-for f in "$LLAMA_BIN" "$MODEL"; do
+for f in "$LLAMA_BIN" "$MODEL" "$CHAT_TEMPLATE_FILE"; do
    if [[ ! -r "$f" ]]; then
        echo "missing or unreadable required artifact: $f" >&2
        exit 1
@@ -121,8 +121,8 @@ if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
    fi
 fi

-# The CIRU package is text-only: no mmproj sidecar. Pass --mmproj only when
-# MMPROJ_PATH points at a readable file (kept for future multimodal packs).
+# The agentionai FAST package includes the model's own vision tower (f16,
+# lossless). Pass --mmproj only when MMPROJ_PATH points at a readable file.
 MMPROJ_ARGS=()
 if [[ -n "${MMPROJ_PATH:-}" && -r "${MMPROJ_PATH:-}" ]]; then
    MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
@@ -138,10 +138,6 @@ exec /usr/bin/env \
    HIP_VISIBLE_DEVICES=0 \
    LD_LIBRARY_PATH="$LIBRARY_PATH" \
    HSA_OVERRIDE_GFX_VERSION="$HSA_OVERRIDE_GFX_VERSION" \
-    GGML_CUDA_Q41_MOE_FORCE_J="${GGML_CUDA_Q41_MOE_FORCE_J:-32}" \
-    GGML_QWEN4EXP_PLE_WORKERS="${GGML_QWEN4EXP_PLE_WORKERS:-16}" \
-    GGML_QWEN4EXP_PLE_STRICT_SHA="${GGML_QWEN4EXP_PLE_STRICT_SHA:-0}" \
-    ROCBLAS_USE_HIPBLASLT="${ROCBLAS_USE_HIPBLASLT:-1}" \
    "$LLAMA_BIN" \
    --model "$MODEL" \
    --alias "$ALIAS" \
@@ -151,7 +147,7 @@ exec /usr/bin/env \
    --port "$LISTEN_PORT" \
    --jinja \
    --device ROCm0 \
-    --gpu-layers "${NGL:-all}" \
+    --gpu-layers "$NGL" \
    --split-mode "$SPLIT_MODE" \
    --parallel "$PARALLEL" \
    --ctx-size "$CTX_SIZE" \
@@ -164,10 +160,7 @@ exec /usr/bin/env \
    --threads "$NUM_THREADS" \
    --threads-batch "$THREAD_BATCH" \
    --ctx-checkpoints "$CTX_CHECKPOINTS" \
-    --checkpoint-min-step "$CHECKPOINT_MIN_STEP" \
    --cache-ram "$CACHE_RAM" \
-    --ple-sidecar "$MODEL_DIR/ple" \
-    --ple-cache-mib "$PLE_CACHE_MIB" \
    --slot-save-path "$SLOT_SAVE_PATH" \
    --cache-prompt \
    --cache-idle-slots \
@@ -181,4 +174,4 @@ exec /usr/bin/env \
    --reasoning-format "$REASONING_FORMAT" \
    --reasoning-budget -1 \
    "${MMPROJ_ARGS[@]}" \
-    "${SPEC_ARGS[@]}"
+    "${SPEC_ARGS[@]}"
@@ -0,0 +1,170 @@
+---
+type: reference
+title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
+description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix, per-head PLE in-GGUF, own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large.
+tags:
+  - llama.cpp
+  - rocm
+  - dankclaw
+  - gfx1151
+  - agentionai
+  - rocmfp4
+  - strix-halo
+  - qwen4exp
+  - mtp
+timestamp: 2026-08-29
+---
+
+# agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
+
+> **Active since 2026-08-29.** Replaced the jcbtc CIRU v1.1 (ciru-ai runtime)
+> deployment on the same slot.
+
+`frank-default-large` (port 7770) serves
+`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
+`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf`) with the author's
+own MTP draft head (`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the
+charlie12345/ROCmFPX `build-qwen38` runtime.
+
+## Why this model
+
+- 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory
+  box, including the 51.2 B-parameter n-gram (PLE) table, which is split per
+  head inside the single GGUF (`v2/` is the joined-table layout; root is the
+  VRAM-resident one and is what we run).
+- imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2
+  (better than AesSedai IQ4_XS / IQ3_S at smaller size).
+- 18-25% faster prompt processing than a mainline quant of similar size
+  (356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth).
+- Ships the model's own f16 vision tower (`mmproj/`) and its own MTP head
+  (~66% draft acceptance, mean accepted length 2.97 of 3 at n-max 3).
+- The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0
+  draft measured worse on this head family (acceptance + throughput).
+
+## Runtime
+
+charlie12345/ROCmFPX `llama-server` built at
+`/home/dank/GitPackages/ROCmFPX/build-qwen38` (HIP, ROCm 7.2.4, gfx1151,
+PR #27742 qwen4exp merged). Upstream llama.cpp cannot load the ROCmFP4
+quant types (`TYPE_100`/`TYPE_101`/FAST variants) — verified:
+
+```
+strings build-qwen38/bin/llama-server | grep Q4_0_ROCMFP4_FAST
+```
+
+CIRU binaries (ciru-ai v1.1) do **not** carry the FAST types; do not point
+this model at the CIRU runtime. The build previously served the kingjones
+STRIX / STRIX_LEAN models; the agentionai-FAST pair is the first use of the
+author's own matched MTP head on this binary.
+
+## Files / download
+
+| Item | Path on Dankclaw | Size |
+|---|---|---|
+| Main | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) |
+| Vision | `…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf` | 904,004,128 B |
+| Draft head | `/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf` | 2,444,519,296 B |
+
+Draft head sha256 (verified against HF LFS):
+`5046d69571bb35c699e19c7a00b36633c2fdfdd06d70482551744ec523a6b590`
+
+Downloaded with a resumable plain `curl -L -C -` (Hugging Face `resolve/main`
+URLs; no auth token needed for public repos). The mmproj lives in an `mmproj/`
+subdirectory of the repo.
+
+## Serving profile
+
+Card sampling table (instruct, non-thinking) mixed with the validated
+build-qwen38 profile:
+
+```
+llama-server \
+  --model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf \
+  --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
+  --host 0.0.0.0 --port 7770 --jinja \
+  --device ROCm0 --gpu-layers 999 \
+  --split-mode none --fit off --flash-attn on \
+  --ctx-size 262144 --parallel 1 \
+  --batch-size 4096 --ubatch-size 4096 \
+  --cache-type-k f16 --cache-type-v f16 \
+  --threads 16 --threads-batch 16 \
+  --ctx-checkpoints 16 --cache-ram 8192 \
+  --cache-prompt --cache-idle-slots --metrics \
+  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
+  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
+  --chat-template-file chat-template.jinja \
+  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
+  --spec-type draft-mtp,ngram-mod \
+  --spec-draft-model Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf \
+  --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.75 \
+  --spec-draft-ngl 999 --spec-draft-device ROCm0 \
+  --spec-draft-type-k f16 --spec-draft-type-v f16
+```
+
+Key choices and deviations from the CIRU deployment:
+
+- **`--fit off`** — auto-fit segfaults on qwen4exp hyper-connection config
+  (same as every Flash-Next deployment).
+- **`--checkpoint-min-step` removed** — build-qwen38 does not implement it
+  (verified via `--help`); only `--ctx-checkpoints` is passed.
+- **No PLE sidecar args** — the n-gram table is in-GGUF (per-head split),
+  not a `ple/` CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the
+  `GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
+  runtime-only and are gone.
+- **`--mmproj` active** — vision tower included (f16, lossless).
+- **MTP n-max 3, no p-split** — author measurements: n-max 3 = 32.4 t/s /
+  0.612 acceptance, adaptive p-split measured worse (28.4 t/s / 0.468).
+  `p-min` stays at the 0.75 default.
+- Context kept at 262144 × 1 slot (same as the CIRU deployment; the author's
+  quick-start is 32768 but the full window is fine on SVM).
+
+## MTP draft head
+
+The draft is agentionai's own head for this exact model:
+`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, architecture
+`qwen4exp` with `blk.48.hc_attn_norm.weight` present (count 1, same shape as
+the CIRU draft that works). The earlier 2026-08-28 failure of this head on
+the *kingjones* STRIX_LEAN main model (`missing tensor blk.0.hc_attn_norm`)
+was a cross-vendor pairing problem, not a defect of the head itself. The
+draft carries a 512-expert MoE, so adaptive/long drafting costs real forward
+passes — keep n-max 3.
+
+## Config files
+
+- Env: `scripts/frank-default-large.env`
+- Serve: `scripts/serve-frank-large.sh`
+- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
+
+## Swap procedure
+
+```bash
+sudo systemctl stop frank-default-large
+# edit scripts/frank-default-large.env + scripts/serve-frank-large.sh, commit, push
+# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-default-large
+sudo systemctl start frank-default-large   # or via deploy.sh
+curl http://192.168.0.143:7770/health
+```
+
+Model load is slow (87 GiB + 2.3 GiB draft): health returns 503 while
+loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.
+
+## Known issues
+
+1. **ROCmFP4 types require the ROCmFPX fork.** Stock llama.cpp and the CIRU
+   runtime reject these files.
+2. **Single 87 GiB file.** Use resumable curl; allow ~10-20 minutes on a
+   good link.
+3. **MTP head must be agentionai's.** EasiiX Q8_0 and other heads are
+   incompatible with this binary's draft expectations for Flash-Next
+   (hyper-connection norm tensor).
+4. **`SVM mapping failed` kernel spam** is a documented symptom, not root
+   cause — investigate before blaming memory pressure.
+
+## Citations
+
+- [agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF)
+- [agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF)
+- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
+- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture) and the ROCmFP4 types
+- [llama.cpp PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742) — qwen4exp support (merged upstream)
+- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support
@@ -1,10 +1,10 @@
 # LLM Fleet — Knowledge Base

 Documentation for the LLM inference fleet on two local machines
-(2026-09-02): FreeToken v0.1.2 on Dankserver (frank-default-small,
+(2026-08-29): FreeToken v0.1.2 on Dankserver (frank-default-small,
 Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw
-(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
-services run the Qwen-Sharp chat template.
+(frank-default-large, agentionai Qwen3.8-Flash-Next ROCmFP4-FAST
+imatrix + MTP draft). Both services run the Qwen-Sharp chat template.

 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -26,10 +26,10 @@ block-fp8, 20% MoE residency, 453k KV / 4 streams]

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        E2[gfx1151 iGPU 64 GiB unified memory]
-        E[charlie12345/ROCmFPX build-kairic (v81)
+        E[charlie12345/ROCmFPX build-qwen38
 frank-default-large, port 7770]
-        F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
-MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
+        F[agentionai Qwen3.8-Flash-Next
+ROCmFP4-FAST imatrix + MTP draft, 262K ctx / 1 slot]
        E2 --> E --> F
    end

@@ -53,7 +53,11 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
  (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[kingjones-strix-rocmfp4]] — active frank-default-large runbook:
+- [[agentionai-fast-imatrix]] — ACTIVE frank-default-large runbook:
+  agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB
+  Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head
+  (ROCmFP4-FAST, 2.28 GiB), build-qwen38 runtime (2026-08-29)
+- [[kingjones-strix-rocmfp4]] — retired frank-default-large runbook:
  kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
  (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, STRIX tier
 - [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
@@ -111,7 +111,10 @@ The LLM inference fleet on the two hosts. Full index:
 - [[freetoken-frank-default-small]] — the active frank-default-small runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver (2026-09-02)
 - [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (restored 2026-09-01)
- [[kingjones-strix-rocmfp4]] — the active frank-default-large runbook:
+- [[agentionai-fast-imatrix]] — the ACTIVE frank-default-large runbook:
+  agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
+  Q4_0_ROCMFP4_FAST) + own MTP draft head, build-qwen38 (2026-08-29)
+- [[kingjones-strix-rocmfp4]] — retired frank-default-large runbook:
  kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B params, 113.47 GiB)
 - [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
 - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw