Frank
8212d4451a
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
217 lines
10 KiB
Markdown
217 lines
10 KiB
Markdown
---
|
||
type: reference
|
||
title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
|
||
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-large.
|
||
tags:
|
||
- llama.cpp
|
||
- rocm
|
||
- dankclaw
|
||
- gfx1151
|
||
- agentionai
|
||
- rocmfp4
|
||
- strix-halo
|
||
- qwen4exp
|
||
- mtp
|
||
timestamp: 2026-08-29
|
||
---
|
||
|
||
# agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
|
||
|
||
> **RETIRED 2026-08-30** — replaced by the kingjones STRIX model on the
|
||
> card's pinned kingjones30/ROCmFPX runtime; see
|
||
> [[kingjones-strix-rocmfp4]] for the active runbook. This page stays as
|
||
> the deployment record: the v2-joined-PLE layout requirement, the MTP
|
||
> head incompatibility (verified 3x), and the QSA F32-mask patch
|
||
> (`scripts/patches/qwen4exp-qsa-f32-mask.patch`) that made
|
||
> `--flash-attn on` work on the charlie build-qwen38 tree — all still
|
||
> true for that binary + model pair, which remains the rollback target.
|
||
|
||
`frank-large` (port 7770) served
|
||
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
|
||
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout)
|
||
with the author's own MTP draft head
|
||
(`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`, never loaded) on the
|
||
charlie12345/ROCmFPX `build-qwen38` runtime, 2026-08-29 → 2026-08-30.
|
||
|
||
## Why this model
|
||
|
||
- 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory
|
||
box, including the 51.2 B-parameter n-gram (PLE) table. REPO `v2/` layout
|
||
is used (joined single `per_layer_token_embd.weight` tensor) because the
|
||
charlie12345/ROCmFPX loader requires it; the root `-ple16` file's per-head
|
||
split only loads in the LaurentZuijdwijk Vulkan fork (see Known issues #2).
|
||
- imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2
|
||
(better than AesSedai IQ4_XS / IQ3_S at smaller size).
|
||
- 18-25% faster prompt processing than a mainline quant of similar size
|
||
(356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth).
|
||
- Ships the model's own f16 vision tower (`mmproj/`).
|
||
- The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0
|
||
draft measured worse on this head family — but it does not load on
|
||
build-qwen38 (Known issues #3).
|
||
|
||
## Runtime
|
||
|
||
charlie12345/ROCmFPX `llama-server` built at
|
||
`/home/dank/GitPackages/ROCmFPX/build-qwen38` (HIP, ROCm 7.2.4, gfx1151,
|
||
PR #27742 qwen4exp merged). Upstream llama.cpp cannot load the ROCmFP4
|
||
quant types (`TYPE_100`/`TYPE_101`/FAST variants) — verified:
|
||
|
||
```
|
||
strings build-qwen38/bin/llama-server | grep Q4_0_ROCMFP4_FAST
|
||
```
|
||
|
||
CIRU binaries (ciru-ai v1.1) do **not** carry the FAST types; do not point
|
||
this model at the CIRU runtime. The build previously served the kingjones
|
||
STRIX / STRIX_LEAN models; the agentionai-FAST pair is the first use of the
|
||
author's own matched MTP head on this binary.
|
||
|
||
## Files / download
|
||
|
||
| Item | Path on Dankclaw | Size |
|
||
|---|---|---|
|
||
| Main (v2 joined) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf` | 93,484,242,816 B (87.06 GiB) |
|
||
| Main (root ple16, unused) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) |
|
||
| Vision | `…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf` | 904,004,128 B |
|
||
| Draft head | `/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf` | 2,444,519,296 B |
|
||
|
||
Draft head sha256 (verified against HF LFS):
|
||
`5046d69571bb35c699e19c7a00b36633c2fdfdd06d70482551744ec523a6b590`
|
||
|
||
Downloaded with a resumable plain `curl -L -C -` (Hugging Face `resolve/main`
|
||
URLs; no auth token needed for public repos). The mmproj lives in an `mmproj/`
|
||
subdirectory of the repo.
|
||
|
||
## Serving profile
|
||
|
||
Card sampling table (instruct, non-thinking) mixed with the validated
|
||
build-qwen38 profile:
|
||
|
||
```
|
||
llama-server \
|
||
--model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf \
|
||
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
|
||
--host 0.0.0.0 --port 7770 --jinja \
|
||
--device ROCm0 --gpu-layers 999 \
|
||
--split-mode none --fit off --flash-attn on \
|
||
--ctx-size 32768 --parallel 1 \
|
||
--batch-size 4096 --ubatch-size 4096 \
|
||
--cache-type-k f16 --cache-type-v f16 \
|
||
--threads 16 --threads-batch 16 \
|
||
--ctx-checkpoints 16 --cache-ram 8192 \
|
||
--cache-prompt --cache-idle-slots --metrics \
|
||
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
|
||
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
|
||
--chat-template-file chat-template.jinja \
|
||
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
|
||
--spec-type ngram-mod
|
||
```
|
||
|
||
Key choices and deviations from the CIRU deployment:
|
||
|
||
- **`--fit off`** — auto-fit segfaults on qwen4exp hyper-connection config
|
||
(same as every Flash-Next deployment).
|
||
- **`--checkpoint-min-step` removed** — build-qwen38 does not implement it
|
||
(verified via `--help`); only `--ctx-checkpoints` is passed.
|
||
- **No PLE sidecar args** — the n-gram table is in-GGUF (joined
|
||
`per_layer_token_embd.weight` from the `v2/` layout), not a `ple/`
|
||
CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the
|
||
`GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
|
||
runtime-only and are gone.
|
||
- **`--mmproj` active** — vision tower included (f16, lossless).
|
||
- **`--flash-attn on` (restored 2026-08-29).** fa on previously aborted graph
|
||
build: the QSA indexer path (layers with attention compression) called
|
||
`ggml_fill` on the F16-cast kq mask (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
|
||
ggml.c:5371, backtrace through `build_attn_qsa`). Fixed in the fork source
|
||
by the QSA F32-mask patch — vendored at
|
||
`scripts/patches/qwen4exp-qsa-f32-mask.patch` (sha256
|
||
`f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f`);
|
||
the binary at `build-qwen38/bin/llama-server` (rebuilt 2026-08-29) already
|
||
carries it. A clean rebuild must re-apply
|
||
`git apply scripts/patches/qwen4exp-qsa-f32-mask.patch` inside
|
||
`/home/dank/GitPackages/ROCmFPX` before `make`. Verified live at
|
||
CTX_SIZE=32768: model loads, health 200, real completion.
|
||
- Context capped at **32768 × 1 slot — the author's own quick-start**. This
|
||
model is fully GPU-resident (87.06 GiB): the PLE n-gram table sits inside
|
||
the GGUF as one tensor, unlike STRIX_LEAN which streamed it from RAM.
|
||
HIP heap is 119.3 GiB; the context buffer is 50.8 GiB at 131072 and
|
||
103 GiB at 262144, so either window OOMs (`cudaMalloc … failed: out of
|
||
memory`, verified live on first graph alloc). At 32768 the buffer is
|
||
~13 GiB (~100 GiB total) and loads clean.
|
||
- MTP is disabled (ngram-mod only) — see the MTP section below.
|
||
|
||
## MTP draft head — INCOMPATIBLE on build-qwen38
|
||
|
||
The author's own matched head (`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-
|
||
FAST-GGUF`, 2.28 GiB, `qwen4exp` arch) does **not** load on this binary:
|
||
|
||
```
|
||
error loading model: missing tensor 'blk.0.hc_attn_norm.weight'
|
||
```
|
||
|
||
The build-qwen38 qwen4exp loader requires a full per-layer
|
||
`blk.<il>.hc_attn_norm.weight` set in the draft (the graph reads all 48
|
||
layers); the head only carries `blk.48.hc_attn_norm.weight` (count 1 — all
|
||
three Flash-Next-era heads examined on this fork have that same partial
|
||
layout). Verified three times, not a cross-vendor artifact:
|
||
|
||
1. EasiiX Q8_0 head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
|
||
2. agentionai FAST head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
|
||
3. agentionai FAST head vs agentionai v2 main (2026-08-29, this deployment)
|
||
— failed identically.
|
||
|
||
The 2.28 GiB head stays on disk; the moment a binary or head appears that
|
||
carries the full per-layer norm set, re-enable with
|
||
`SPEC_TYPE=draft-mtp,ngram-mod`, `--spec-draft-n-max 3` (author-measured
|
||
best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
|
||
|
||
## Config files
|
||
|
||
- Env: `scripts/frank-large.env`
|
||
- Serve: `scripts/serve-frank-large.sh`
|
||
- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
|
||
- Fork patch: `scripts/patches/qwen4exp-qsa-f32-mask.patch` (re-apply after
|
||
any clean fork rebuild — see Serving profile)
|
||
|
||
## Swap procedure
|
||
|
||
```bash
|
||
sudo systemctl stop frank-large
|
||
# edit scripts/frank-large.env + scripts/serve-frank-large.sh, commit, push
|
||
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-large
|
||
sudo systemctl start frank-large # or via deploy.sh
|
||
curl http://192.168.0.143:7770/health
|
||
```
|
||
|
||
Model load is slow (87 GiB + 2.3 GiB draft): health returns 503 while
|
||
loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.
|
||
|
||
## Known issues
|
||
|
||
1. **ROCmFP4 types require the ROCmFPX fork.** Stock llama.cpp and the CIRU
|
||
runtime reject these files.
|
||
2. **Use the `v2/` joined layout, NOT the root `-ple16` split.** This binary's
|
||
qwen4exp loader (qwen4exp.cpp:104-108) unconditionally requires a single
|
||
`per_layer_token_embd.weight` tensor and GGML_ASSERTs "qwen4exp is missing
|
||
the PLE n-gram table" against the per-head split. The root file only loads
|
||
in the LaurentZuijdwijk Vulkan fork (`vulkan-qwen4exp-rocmfpx`). The v2
|
||
file is byte-identical in weights, ~5 KB larger header, loads fully into
|
||
SVM unified memory (no `--ngram-on-disk` needed — kingjones STRIX ran a
|
||
50 GiB joined PLE table the same way).
|
||
3. **No MTP draft head loads on build-qwen38.** The qwen4exp loader
|
||
requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; every
|
||
Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries
|
||
only `blk.48`. Spec decoding is ngram-mod only. See the MTP section.
|
||
4. **`--flash-attn on` (restored 2026-08-29).** fa on aborts graph build
|
||
without the QSA F32-mask patch — see Serving profile (fix + sha256
|
||
vendored there).
|
||
5. **`SVM mapping failed` kernel spam** is a documented symptom, not root
|
||
cause — investigate before blaming memory pressure.
|
||
|
||
## Citations
|
||
|
||
- [agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF)
|
||
- [agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF)
|
||
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
|
||
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture) and the ROCmFP4 types
|
||
- [llama.cpp PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742) — qwen4exp support (merged upstream)
|
||
- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support |