Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
189 lines
7.8 KiB
Markdown
189 lines
7.8 KiB
Markdown
---
|
|
type: reference
|
|
title: caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
|
|
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-dragoon.
|
|
tags:
|
|
- llama.cpp
|
|
- rocm
|
|
- dankclaw
|
|
- gfx1151
|
|
- rocmi4
|
|
- mtp
|
|
- iu4
|
|
timestamp: 2026-08-25
|
|
---
|
|
|
|
# caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
|
|
|
|
> **RETIRED 2026-08-25.** `frank-dragoon` served this model from 2026-08-18
|
|
> until 2026-08-25. It replaced the kingjones777 ROCmFP4 STRIX MTP payload
|
|
> on the same port and the same Strix Halo iGPU, running the new
|
|
> charlie12345/ROCmFPX ROCmI4 W4A4 build (see [[kingjones-strix-mtp]] for
|
|
> the page it replaced), and was itself replaced by
|
|
> [[kingjones-strix-rocmfp4]].
|
|
|
|
`frank-dragoon` (port 7770) serves
|
|
`cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
|
|
`eemin` namespace) from the **charlie12345/ROCmFPX** llama.cpp fork built with
|
|
the opt-in `ROCmI4 W4A4` path. The model is `Q4_0_ROCMI4`: signed four-bit
|
|
weights packed two per byte, quantized from the BF16 Qwen3.8-27B and carrying
|
|
its **embedded NextN/MTP head** in the main GGUF — so there is no separate
|
|
draft file. The native gfx1151 `v_wmma_i32_16x16x16_iu4` matrix instruction
|
|
batch-verifies the MTP proposals for higher decode throughput.
|
|
|
|
## What This Is
|
|
|
|
**ROCmI4** is the GGUF weight format (signed 4-bit codes). **IU4** is the AMD
|
|
WMMA instruction spelling that consumes them. **W4A4** means four-bit weights
|
|
and four-bit activations during the accelerated matrix multiply. Ordinary
|
|
one-token decode stays memory-bandwidth-bound (MMVQ); batched prompt processing
|
|
and batched MTP target verification use the IU4 MMQ path. W4A4 is a lossy
|
|
activation-quant speed path — treat it as a speed/quality tradeoff.
|
|
|
|
The model is a conversion/quantization of `Qwen/Qwen3.8-27B` (Unsloth lineage,
|
|
Apache-2.0), not a retrained model. It reports architecture `qwen35`
|
|
(M-RoPE), ~27.32B params, native 262144 context, 866 tensors, ftype 108.
|
|
|
|
## Model Files
|
|
|
|
| File | Size | SHA-256 |
|
|
|------|------|---------|
|
|
| `Qwen3.8-27B-Q4_0_ROCMI4.gguf` | 13.50 GiB (14,534,384,384 B) | `ad51ecbee55930ababc629c377943a7bbe50f8aafcf32340c7cdae6fa6b7bb3c` |
|
|
| `mmproj-BF16.gguf` (vision, optional) | 0.87 GiB (931,146,432 B) | `83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53` |
|
|
|
|
Store under `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/`. The vision
|
|
projector is optional for text-only use.
|
|
|
|
## Runtime Build
|
|
|
|
The stock llama.cpp build does not understand `Q4_0_ROCMI4` or the qualified
|
|
IU4/MTP path. Use charlie12345/ROCmFPX at the pinned commit, built HIP-only
|
|
(so the combined HIP/Vulkan scheduler cannot route the ROCmI4 tensors to
|
|
Vulkan, which has no shader for this type) with the W4A4 option on.
|
|
|
|
| Artifact | Path |
|
|
|----------|------|
|
|
| Fork + pinned commit | `/home/dank/rocmi4-qwen38/ROCmFPX` @ `c49ebdbd5c9f01ec242369f9e7f7967855f80cba` |
|
|
| Build | `cmake -S . -B build-rocmi4-w4a4 -G Ninja -DGGML_HIP=ON -DGGML_VULKAN=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_HIP_ROCMI4_W4A4=ON -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF` |
|
|
| Binary | `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (version 257, `c49ebdb`) |
|
|
| ROCm | system `/opt/rocm-7.2.4`, amdclang 22 |
|
|
|
|
Build a separate directory so the exact-int8 path (the default branch) stays
|
|
available. Rollback: disable `-DGGML_HIP_ROCMI4_W4A4=ON` and rebuild — no model
|
|
conversion or data change needed. The build prints
|
|
`ROCmI4 W4A4: enabled for device 0` when the accelerated path is active.
|
|
|
|
## Model Download
|
|
|
|
Downloaded from the public
|
|
`eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF` repo (the card's download section still
|
|
names the still-private `cafonz/…`; use `eemin`). Verify against the repo's
|
|
`checksums.sha256` after download:
|
|
|
|
```
|
|
cd /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP
|
|
sha256sum -c checksums.sha256
|
|
```
|
|
|
|
## Launch Profile
|
|
|
|
The qualified MTP profile (from the model card and PR #94, verified on the
|
|
live service): strict Qwen MTP, backend sampling, `n_max 4`, `p_min 0.60`.
|
|
|
|
```mermaid
|
|
graph TD
|
|
A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
|
|
B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
|
|
C --> D[strict Qwen MTP<br/>n_max 4 / p_min 0.60<br/>backend sampling]
|
|
D --> E[Sharp chat template<br/>reasoning deepseek]
|
|
```
|
|
|
|
Key flags (see `scripts/serve-frank-dragoon.sh`):
|
|
|
|
```
|
|
--spec-type draft-mtp --spec-mtp-strict-qwen
|
|
--spec-draft-device ROCm0 --spec-draft-ngl all
|
|
--spec-draft-type-k f16 --spec-draft-type-v f16
|
|
--spec-draft-n-max 4 --spec-draft-n-min 0
|
|
--spec-draft-p-min 0.60 --spec-draft-backend-sampling
|
|
```
|
|
|
|
Because the MTP head is **embedded**, no `--model-draft` is passed and the
|
|
draft uses all layers on ROCm0. `--spec-mtp-strict-qwen` preserves the target
|
|
path's greedy decisions; it does not remove the W4A4 activation approximation.
|
|
|
|
### Serving settings
|
|
|
|
- 65536 context (192K below the native 262144 maximum), 1 slot, f16 KV,
|
|
context checkpoints, prompt cache. The lower limit leaves headroom for the
|
|
model plus KV cache in the host's unified-memory budget. Concurrent requests
|
|
queue; the server never preempts a running generation.
|
|
- `-t 16 -tb 32`, batch 512 / ubatch 256 (the qualified W4A4 profile).
|
|
- `--temp 0.7 --top-p 0.95`, reasoning on (`deepseek` puts `<think>` into
|
|
`message.reasoning_content`).
|
|
- Qwen-Sharp chat template (`--chat-template-file`), `enable_thinking` +
|
|
`preserve_thinking` on, preserving the peg-native + deepseek shape the Pi
|
|
agent expects.
|
|
- Strix Halo unified memory + native gfx1151 detection: `HSA_OVERRIDE_GFX_VERSION=11.5.1`,
|
|
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`.
|
|
|
|
### Service
|
|
|
|
- Unit: `frank-dragoon.service`, port 7770
|
|
- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
|
|
|
|
## Tuning MTP
|
|
|
|
MTP gains are content-dependent: predictable code, structured data, and lists
|
|
accept longer drafts than creative prose. The active conservative profile is
|
|
`n4 / p0.60`; the previously qualified starting point was `n16 / p0.60`.
|
|
|
|
## ubatch Benchmark
|
|
|
|
Decode throughput depends on the ubatch (per-eval token batch). Run the sweep
|
|
on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch:
|
|
|
|
```
|
|
sudo systemctl stop frank-dragoon
|
|
./scripts/bench-ubatch.sh
|
|
sudo systemctl start frank-dragoon
|
|
```
|
|
|
|
The script stops `frank-dragoon` for a clean GPU, boots one server
|
|
instance per ubatch on a scratch port, reads authoritative `print_timing` tok/s,
|
|
and restarts the service. See [[frank-dragoon-benchmark]].
|
|
|
|
## Known Issues
|
|
|
|
1. **gfx1151 only** for the W4A4 path. Other GPUs fall back to exact int8 MMQ.
|
|
2. **Memory pressure:** the model is 13.5 GiB. Under `--no-mmap` with full
|
|
offload it needs the whole tensor set at once; leave room in the 64 GiB
|
|
unified pool (rocm-smi figure). Do not run a bench with the live service up
|
|
or it OOMs at load.
|
|
3. **`Q4_0_ROCMI4` + MTP flags need the ROCmFPX fork**; stock llama.cpp may
|
|
reject the file or drop the optimized path.
|
|
4. **W4A4 is lossy** (+~5.4% perplexity vs exact int8). Opt-in speed path.
|
|
|
|
## Measured Results (qualification)
|
|
|
|
From the model card, gfx1151, pinned build, W4A4 + strict MTP-16:
|
|
|
|
| Result | Value |
|
|
|--------|-------|
|
|
| HumanEval pilot, W4A4 IU4 (mean) | 49.40 tok/s |
|
|
| Full HumanEval, W4A4 (mean / median) | 44.39 / 45.23 tok/s |
|
|
| HumanEval / HumanEval+ pass@1 | 94.5% / 91.5% |
|
|
| Non-speculative tg128 | ~13.8 tok/s |
|
|
|
|
Local smoke test 2026-08-25 on the iGPU: model loaded, strict-MTP active,
|
|
draft acceptance 0.74 (mean 7.67), generation ~25 tok/s at a small output.
|
|
The ubatch sweep above gives the production ubatch choice.
|
|
|
|
## See Also
|
|
|
|
- [[deployment]] — deploy and update process
|
|
- [[kingjones-strix-mtp]] — the retired ROCmFP4 predecessor on this slot
|
|
- [[ciru-activefpx-promptforge]] — the retired v3 CIRU PromptForge page
|
|
- [[Dankclaw System Specs]] — the box
|
|
- [[frank-dragoon-benchmark]] — backend/spec benchmark history
|