frank-default-large: swap to agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix + own MTP draft
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB, Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF — fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST (2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3). Vision tower (f16) loaded via mmproj/. Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) — the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1 deployment (ciru-ai runtime). Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0, presence 1.5) mixed with the validated build-qwen38 serving profile: 262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on, cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx checkpoints (no --checkpoint-min-step — unsupported on this binary), --fit off (auto-fit segfaults on qwen4exp hyper-connection config), --spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured worse per author card), p-min 0.75. Qwen-Sharp chat template kept; reasoning stays in DeepSeek format. The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN model was a cross-vendor pairing problem: the head carries blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
This commit is contained in:
Frank
@@ -0,0 +1,170 @@
|
||||
---
|
||||
type: reference
|
||||
title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
|
||||
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix, per-head PLE in-GGUF, own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- agentionai
|
||||
- rocmfp4
|
||||
- strix-halo
|
||||
- qwen4exp
|
||||
- mtp
|
||||
timestamp: 2026-08-29
|
||||
---
|
||||
|
||||
# agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
|
||||
|
||||
> **Active since 2026-08-29.** Replaced the jcbtc CIRU v1.1 (ciru-ai runtime)
|
||||
> deployment on the same slot.
|
||||
|
||||
`frank-default-large` (port 7770) serves
|
||||
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
|
||||
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf`) with the author's
|
||||
own MTP draft head (`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the
|
||||
charlie12345/ROCmFPX `build-qwen38` runtime.
|
||||
|
||||
## Why this model
|
||||
|
||||
- 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory
|
||||
box, including the 51.2 B-parameter n-gram (PLE) table, which is split per
|
||||
head inside the single GGUF (`v2/` is the joined-table layout; root is the
|
||||
VRAM-resident one and is what we run).
|
||||
- imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2
|
||||
(better than AesSedai IQ4_XS / IQ3_S at smaller size).
|
||||
- 18-25% faster prompt processing than a mainline quant of similar size
|
||||
(356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth).
|
||||
- Ships the model's own f16 vision tower (`mmproj/`) and its own MTP head
|
||||
(~66% draft acceptance, mean accepted length 2.97 of 3 at n-max 3).
|
||||
- The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0
|
||||
draft measured worse on this head family (acceptance + throughput).
|
||||
|
||||
## Runtime
|
||||
|
||||
charlie12345/ROCmFPX `llama-server` built at
|
||||
`/home/dank/GitPackages/ROCmFPX/build-qwen38` (HIP, ROCm 7.2.4, gfx1151,
|
||||
PR #27742 qwen4exp merged). Upstream llama.cpp cannot load the ROCmFP4
|
||||
quant types (`TYPE_100`/`TYPE_101`/FAST variants) — verified:
|
||||
|
||||
```
|
||||
strings build-qwen38/bin/llama-server | grep Q4_0_ROCMFP4_FAST
|
||||
```
|
||||
|
||||
CIRU binaries (ciru-ai v1.1) do **not** carry the FAST types; do not point
|
||||
this model at the CIRU runtime. The build previously served the kingjones
|
||||
STRIX / STRIX_LEAN models; the agentionai-FAST pair is the first use of the
|
||||
author's own matched MTP head on this binary.
|
||||
|
||||
## Files / download
|
||||
|
||||
| Item | Path on Dankclaw | Size |
|
||||
|---|---|---|
|
||||
| Main | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) |
|
||||
| Vision | `…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf` | 904,004,128 B |
|
||||
| Draft head | `/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf` | 2,444,519,296 B |
|
||||
|
||||
Draft head sha256 (verified against HF LFS):
|
||||
`5046d69571bb35c699e19c7a00b36633c2fdfdd06d70482551744ec523a6b590`
|
||||
|
||||
Downloaded with a resumable plain `curl -L -C -` (Hugging Face `resolve/main`
|
||||
URLs; no auth token needed for public repos). The mmproj lives in an `mmproj/`
|
||||
subdirectory of the repo.
|
||||
|
||||
## Serving profile
|
||||
|
||||
Card sampling table (instruct, non-thinking) mixed with the validated
|
||||
build-qwen38 profile:
|
||||
|
||||
```
|
||||
llama-server \
|
||||
--model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf \
|
||||
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
|
||||
--host 0.0.0.0 --port 7770 --jinja \
|
||||
--device ROCm0 --gpu-layers 999 \
|
||||
--split-mode none --fit off --flash-attn on \
|
||||
--ctx-size 262144 --parallel 1 \
|
||||
--batch-size 4096 --ubatch-size 4096 \
|
||||
--cache-type-k f16 --cache-type-v f16 \
|
||||
--threads 16 --threads-batch 16 \
|
||||
--ctx-checkpoints 16 --cache-ram 8192 \
|
||||
--cache-prompt --cache-idle-slots --metrics \
|
||||
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
|
||||
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
|
||||
--chat-template-file chat-template.jinja \
|
||||
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
|
||||
--spec-type draft-mtp,ngram-mod \
|
||||
--spec-draft-model Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf \
|
||||
--spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.75 \
|
||||
--spec-draft-ngl 999 --spec-draft-device ROCm0 \
|
||||
--spec-draft-type-k f16 --spec-draft-type-v f16
|
||||
```
|
||||
|
||||
Key choices and deviations from the CIRU deployment:
|
||||
|
||||
- **`--fit off`** — auto-fit segfaults on qwen4exp hyper-connection config
|
||||
(same as every Flash-Next deployment).
|
||||
- **`--checkpoint-min-step` removed** — build-qwen38 does not implement it
|
||||
(verified via `--help`); only `--ctx-checkpoints` is passed.
|
||||
- **No PLE sidecar args** — the n-gram table is in-GGUF (per-head split),
|
||||
not a `ple/` CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the
|
||||
`GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
|
||||
runtime-only and are gone.
|
||||
- **`--mmproj` active** — vision tower included (f16, lossless).
|
||||
- **MTP n-max 3, no p-split** — author measurements: n-max 3 = 32.4 t/s /
|
||||
0.612 acceptance, adaptive p-split measured worse (28.4 t/s / 0.468).
|
||||
`p-min` stays at the 0.75 default.
|
||||
- Context kept at 262144 × 1 slot (same as the CIRU deployment; the author's
|
||||
quick-start is 32768 but the full window is fine on SVM).
|
||||
|
||||
## MTP draft head
|
||||
|
||||
The draft is agentionai's own head for this exact model:
|
||||
`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, architecture
|
||||
`qwen4exp` with `blk.48.hc_attn_norm.weight` present (count 1, same shape as
|
||||
the CIRU draft that works). The earlier 2026-08-28 failure of this head on
|
||||
the *kingjones* STRIX_LEAN main model (`missing tensor blk.0.hc_attn_norm`)
|
||||
was a cross-vendor pairing problem, not a defect of the head itself. The
|
||||
draft carries a 512-expert MoE, so adaptive/long drafting costs real forward
|
||||
passes — keep n-max 3.
|
||||
|
||||
## Config files
|
||||
|
||||
- Env: `scripts/frank-default-large.env`
|
||||
- Serve: `scripts/serve-frank-large.sh`
|
||||
- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
|
||||
|
||||
## Swap procedure
|
||||
|
||||
```bash
|
||||
sudo systemctl stop frank-default-large
|
||||
# edit scripts/frank-default-large.env + scripts/serve-frank-large.sh, commit, push
|
||||
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-default-large
|
||||
sudo systemctl start frank-default-large # or via deploy.sh
|
||||
curl http://192.168.0.143:7770/health
|
||||
```
|
||||
|
||||
Model load is slow (87 GiB + 2.3 GiB draft): health returns 503 while
|
||||
loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.
|
||||
|
||||
## Known issues
|
||||
|
||||
1. **ROCmFP4 types require the ROCmFPX fork.** Stock llama.cpp and the CIRU
|
||||
runtime reject these files.
|
||||
2. **Single 87 GiB file.** Use resumable curl; allow ~10-20 minutes on a
|
||||
good link.
|
||||
3. **MTP head must be agentionai's.** EasiiX Q8_0 and other heads are
|
||||
incompatible with this binary's draft expectations for Flash-Next
|
||||
(hyper-connection norm tensor).
|
||||
4. **`SVM mapping failed` kernel spam** is a documented symptom, not root
|
||||
cause — investigate before blaming memory pressure.
|
||||
|
||||
## Citations
|
||||
|
||||
- [agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF)
|
||||
- [agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF)
|
||||
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
|
||||
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture) and the ROCmFP4 types
|
||||
- [llama.cpp PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742) — qwen4exp support (merged upstream)
|
||||
- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support
|
||||
@@ -1,10 +1,10 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-09-02): FreeToken v0.1.2 on Dankserver (frank-default-small,
|
||||
(2026-08-29): FreeToken v0.1.2 on Dankserver (frank-default-small,
|
||||
Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
|
||||
services run the Qwen-Sharp chat template.
|
||||
(frank-default-large, agentionai Qwen3.8-Flash-Next ROCmFP4-FAST
|
||||
imatrix + MTP draft). Both services run the Qwen-Sharp chat template.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -26,10 +26,10 @@ block-fp8, 20% MoE residency, 453k KV / 4 streams]
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[charlie12345/ROCmFPX build-kairic (v81)
|
||||
E[charlie12345/ROCmFPX build-qwen38
|
||||
frank-default-large, port 7770]
|
||||
F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
|
||||
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
F[agentionai Qwen3.8-Flash-Next
|
||||
ROCmFP4-FAST imatrix + MTP draft, 262K ctx / 1 slot]
|
||||
E2 --> E --> F
|
||||
end
|
||||
|
||||
@@ -53,7 +53,11 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[kingjones-strix-rocmfp4]] — active frank-default-large runbook:
|
||||
- [[agentionai-fast-imatrix]] — ACTIVE frank-default-large runbook:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB
|
||||
Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head
|
||||
(ROCmFP4-FAST, 2.28 GiB), build-qwen38 runtime (2026-08-29)
|
||||
- [[kingjones-strix-rocmfp4]] — retired frank-default-large runbook:
|
||||
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, STRIX tier
|
||||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||||
|
||||
Reference in New Issue
Block a user