frank-default-large: swap to caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP
Replace the retired kingjones ROCmFP4 STRIX MTP payload on Dankclaw with cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF (eemin namespace, public), served by the charlie12345/ROCmFPX llama.cpp build (ROCmI4 W4A4, c49ebdbd) on the Strix Halo iGPU. Q4_0_ROCMI4 with an embedded NextN/MTP head (no separate draft file); strict-Qwen MTP n16/p0.60/backend sampling on ROCm0. Binary /home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server; model dir /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP. Updated serve script + env, ubatch bench, large-config bats test, docs, AGENTS.md.
This commit is contained in:
Frank committed
1 parent
7a8500f6fa
commit
7ca9dc248e
10 files changed
+626
-131
No files matched your search
+20
-15
@@ -18,7 +18,7 @@ repository to a running service.
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q6_K, native MTP draft, selected MoE experts offloaded via --override-tensor) | frank-default-small | 7770 |
|
||||
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (Qwen3.8-27B-Uncensored-ROCmFP4 Q6 AGENT, standalone MTP Q4_0 head, 262K ctx / 1 slot) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | charlie12345/ROCmFPX llama.cpp (caf-eemin Qwen3.8-27B ROCmI4 Q4_0, embedded MTP, ROCmI4 W4A4, 262K ctx / 1 slot) | frank-default-large | 7770 |
|
||||
|
||||
## Workflow
|
||||
|
||||
@@ -137,24 +137,29 @@ from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]].
|
||||
|
||||
### Dankclaw — frank-default-large
|
||||
|
||||
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
|
||||
`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` (Q6_0 AGENT, dense)
|
||||
under the alias `frank-default-large` on port 7770. It runs
|
||||
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
|
||||
7.15 SDK in the retired ling3-ciru venv) with a standalone MTP Q4_0 draft
|
||||
head (`--model-draft` and `--spec-type draft-mtp`), f16 KV, full 262144
|
||||
context (1 slot. Concurrent requests queue; the server never preempts a
|
||||
running generation), batch 4096 / ubatch 4096, 16 context checkpoints,
|
||||
8 GiB prompt cache, and reasoning in deepseek format. The unit file is
|
||||
The active service on Dankclaw is the charlie12345/ROCmFPX llama.cpp runtime
|
||||
(with the opt-in `ROCmI4 W4A4` path) serving `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF`
|
||||
(`eemin` namespace, `Q4_0_ROCMI4`, embedded NextN/MTP head) under the alias
|
||||
`frank-default-large` on port 7770. It runs
|
||||
`/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (built
|
||||
against system ROCm 7.2.4 / amdclang 22) with a **strict Qwen MTP** draft
|
||||
(`--spec-type draft-mtp --spec-mtp-strict-qwen`) running on ROCm0 with all
|
||||
layers (`--spec-draft-ngl all`) — no separate draft file, because the MTP head
|
||||
is embedded in the main GGUF. K/V are f16, full 262144 context (1 slot,
|
||||
concurrent requests queue, the server never preempts a running generation),
|
||||
`-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch is swappable; sweep it on 8k
|
||||
context with `scripts/bench-ubatch-rocmi4.sh`), 16 context checkpoints, 8 GiB
|
||||
prompt cache, and reasoning in deepseek format. The unit file is
|
||||
`scripts/frank-default-large.service`. It launches through
|
||||
`scripts/serve-frank-large.sh`, which reads tunables from
|
||||
`scripts/frank-default-large.env`, applying the Qwen-Sharp chat template.
|
||||
See [[kingjones-strix-mtp]].
|
||||
See [[eemin-rocmi4-mtp]].
|
||||
|
||||
> The kingjones Q6 AGENT model has served the slot since 2026-08-24; the
|
||||
> v3 PromptForge payload is retired on this slot.
|
||||
> Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was
|
||||
> retired the same day. See [[dflash2-large-vulkan]] for the archived runbook.
|
||||
> The kingjones ROCmFP4 STRIX MTP model served the slot from 2026-08-24; it
|
||||
> was replaced by the caf-eemin ROCmI4 embedded-MTP model on 2026-08-25. The
|
||||
> v3 PromptForge payload and the CIRU ROCmFPX runtime are retired on this slot.
|
||||
> The intermediate DFlash2 Vulkan deployment was retired the same day (2026-08-21).
|
||||
> See [[dflash2-large-vulkan]] for the archived runbook.
|
||||
|
||||
## Validation
|
||||
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
---
|
||||
type: reference
|
||||
title: caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
|
||||
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-default-large.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- rocmi4
|
||||
- mtp
|
||||
- iu4
|
||||
timestamp: 2026-08-25
|
||||
---
|
||||
|
||||
# caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
|
||||
|
||||
> **Active 2026-08-25.** `frank-default-large` serves this model. It replaced
|
||||
> the kingjones777 ROCmFP4 STRIX MTP payload on the same port and the same
|
||||
> Strix Halo iGPU, running the new charlie12345/ROCmFPX ROCmI4 W4A4 build. See
|
||||
> [[kingjones-strix-mtp]] for the retired page.
|
||||
|
||||
`frank-default-large` (port 7770) serves
|
||||
`cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
|
||||
`eemin` namespace) from the **charlie12345/ROCmFPX** llama.cpp fork built with
|
||||
the opt-in `ROCmI4 W4A4` path. The model is `Q4_0_ROCMI4`: signed four-bit
|
||||
weights packed two per byte, quantized from the BF16 Qwen3.8-27B and carrying
|
||||
its **embedded NextN/MTP head** in the main GGUF — so there is no separate
|
||||
draft file. The native gfx1151 `v_wmma_i32_16x16x16_iu4` matrix instruction
|
||||
batch-verifies the MTP proposals for higher decode throughput.
|
||||
|
||||
## What This Is
|
||||
|
||||
**ROCmI4** is the GGUF weight format (signed 4-bit codes). **IU4** is the AMD
|
||||
WMMA instruction spelling that consumes them. **W4A4** means four-bit weights
|
||||
and four-bit activations during the accelerated matrix multiply. Ordinary
|
||||
one-token decode stays memory-bandwidth-bound (MMVQ); batched prompt processing
|
||||
and batched MTP target verification use the IU4 MMQ path. W4A4 is a lossy
|
||||
activation-quant speed path — treat it as a speed/quality tradeoff.
|
||||
|
||||
The model is a conversion/quantization of `Qwen/Qwen3.8-27B` (Unsloth lineage,
|
||||
Apache-2.0), not a retrained model. It reports architecture `qwen35`
|
||||
(M-RoPE), ~27.32B params, native 262144 context, 866 tensors, ftype 108.
|
||||
|
||||
## Model Files
|
||||
|
||||
| File | Size | SHA-256 |
|
||||
|------|------|---------|
|
||||
| `Qwen3.8-27B-Q4_0_ROCMI4.gguf` | 13.50 GiB (14,534,384,384 B) | `ad51ecbee55930ababc629c377943a7bbe50f8aafcf32340c7cdae6fa6b7bb3c` |
|
||||
| `mmproj-BF16.gguf` (vision, optional) | 0.87 GiB (931,146,432 B) | `83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53` |
|
||||
|
||||
Store under `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/`. The vision
|
||||
projector is optional for text-only use.
|
||||
|
||||
## Runtime Build
|
||||
|
||||
The stock llama.cpp build does not understand `Q4_0_ROCMI4` or the qualified
|
||||
IU4/MTP path. Use charlie12345/ROCmFPX at the pinned commit, built HIP-only
|
||||
(so the combined HIP/Vulkan scheduler cannot route the ROCmI4 tensors to
|
||||
Vulkan, which has no shader for this type) with the W4A4 option on.
|
||||
|
||||
| Artifact | Path |
|
||||
|----------|------|
|
||||
| Fork + pinned commit | `/home/dank/rocmi4-qwen38/ROCmFPX` @ `c49ebdbd5c9f01ec242369f9e7f7967855f80cba` |
|
||||
| Build | `cmake -S . -B build-rocmi4-w4a4 -G Ninja -DGGML_HIP=ON -DGGML_VULKAN=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_HIP_ROCMI4_W4A4=ON -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF` |
|
||||
| Binary | `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (version 257, `c49ebdb`) |
|
||||
| ROCm | system `/opt/rocm-7.2.4`, amdclang 22 |
|
||||
|
||||
Build a separate directory so the exact-int8 path (the default branch) stays
|
||||
available. Rollback: disable `-DGGML_HIP_ROCMI4_W4A4=ON` and rebuild — no model
|
||||
conversion or data change needed. The build prints
|
||||
`ROCmI4 W4A4: enabled for device 0` when the accelerated path is active.
|
||||
|
||||
## Model Download
|
||||
|
||||
Downloaded from the public
|
||||
`eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF` repo (the card's download section still
|
||||
names the still-private `cafonz/…`; use `eemin`). Verify against the repo's
|
||||
`checksums.sha256` after download:
|
||||
|
||||
```
|
||||
cd /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP
|
||||
sha256sum -c checksums.sha256
|
||||
```
|
||||
|
||||
## Launch Profile
|
||||
|
||||
The qualified MTP profile (from the model card and PR #94, verified on the
|
||||
live service): strict Qwen MTP, backend sampling, `n_max 16`, `p_min 0.60`.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
|
||||
B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
|
||||
C --> D[strict Qwen MTP<br/>n_max 16 / p_min 0.60<br/>backend sampling]
|
||||
D --> E[Sharp chat template<br/>reasoning deepseek]
|
||||
```
|
||||
|
||||
Key flags (see `scripts/serve-frank-large.sh`):
|
||||
|
||||
```
|
||||
--spec-type draft-mtp --spec-mtp-strict-qwen
|
||||
--spec-draft-device ROCm0 --spec-draft-ngl all
|
||||
--spec-draft-type-k f16 --spec-draft-type-v f16
|
||||
--spec-draft-n-max 16 --spec-draft-n-min 0
|
||||
--spec-draft-p-min 0.60 --spec-draft-backend-sampling
|
||||
```
|
||||
|
||||
Because the MTP head is **embedded**, no `--model-draft` is passed and the
|
||||
draft uses all layers on ROCm0. `--spec-mtp-strict-qwen` preserves the target
|
||||
path's greedy decisions; it does not remove the W4A4 activation approximation.
|
||||
|
||||
### Serving settings
|
||||
|
||||
- 262144 context (native max), 1 slot, f16 KV, context checkpoints, prompt
|
||||
cache. Concurrent requests queue; the server never preempts a running
|
||||
generation.
|
||||
- `-t 16 -tb 32`, batch 512 / ubatch 256 (the qualified W4A4 profile).
|
||||
- `--temp 0.7 --top-p 0.95`, reasoning on (`deepseek` puts `<think>` into
|
||||
`message.reasoning_content`).
|
||||
- Qwen-Sharp chat template (`--chat-template-file`), `enable_thinking` +
|
||||
`preserve_thinking` on, preserving the peg-native + deepseek shape the Pi
|
||||
agent expects.
|
||||
- Strix Halo unified memory + native gfx1151 detection: `HSA_OVERRIDE_GFX_VERSION=11.5.1`,
|
||||
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`.
|
||||
|
||||
### Service
|
||||
|
||||
- Unit: `frank-default-large.service`, port 7770
|
||||
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
|
||||
|
||||
## Tuning MTP
|
||||
|
||||
MTP gains are content-dependent: predictable code, structured data, and lists
|
||||
accept longer drafts than creative prose. The qualified start is `n16 / p0.60`.
|
||||
If acceptance or stability is poor on another workload, lower
|
||||
`--spec-draft-n-max` to 4 or 6 and remeasure.
|
||||
|
||||
## ubatch Benchmark
|
||||
|
||||
Decode throughput depends on the ubatch (per-eval token batch). Run the sweep
|
||||
on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch:
|
||||
|
||||
```
|
||||
STOPSVC=1 ./scripts/bench-ubatch-rocmi4.sh
|
||||
```
|
||||
|
||||
The script stops `frank-default-large` for a clean GPU, boots one server
|
||||
instance per ubatch on a scratch port, reads authoritative `print_timing` tok/s,
|
||||
and restarts the service. See [[frank-default-large-benchmark]].
|
||||
|
||||
## Known Issues
|
||||
|
||||
1. **gfx1151 only** for the W4A4 path. Other GPUs fall back to exact int8 MMQ.
|
||||
2. **Memory pressure:** the model is 13.5 GiB. Under `--no-mmap` with full
|
||||
offload it needs the whole tensor set at once; leave room in the 64 GiB
|
||||
unified pool (rocm-smi figure). Do not run a bench with the live service up
|
||||
or it OOMs at load.
|
||||
3. **`Q4_0_ROCMI4` + MTP flags need the ROCmFPX fork**; stock llama.cpp may
|
||||
reject the file or drop the optimized path.
|
||||
4. **W4A4 is lossy** (+~5.4% perplexity vs exact int8). Opt-in speed path.
|
||||
|
||||
## Measured Results (qualification)
|
||||
|
||||
From the model card, gfx1151, pinned build, W4A4 + strict MTP-16:
|
||||
|
||||
| Result | Value |
|
||||
|--------|-------|
|
||||
| HumanEval pilot, W4A4 IU4 (mean) | 49.40 tok/s |
|
||||
| Full HumanEval, W4A4 (mean / median) | 44.39 / 45.23 tok/s |
|
||||
| HumanEval / HumanEval+ pass@1 | 94.5% / 91.5% |
|
||||
| Non-speculative tg128 | ~13.8 tok/s |
|
||||
|
||||
Local smoke test 2026-08-25 on the iGPU: model loaded, strict-MTP active,
|
||||
draft acceptance 0.74 (mean 7.67), generation ~25 tok/s at a small output.
|
||||
The ubatch sweep above gives the production ubatch choice.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[kingjones-strix-mtp]] — the retired ROCmFP4 predecessor on this slot
|
||||
- [[ciru-activefpx-promptforge]] — the retired v3 CIRU PromptForge page
|
||||
- [[Dankclaw System Specs]] — the box
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark history
|
||||
+6
-2
@@ -48,8 +48,12 @@ MTP draft + Sharp template, 262K ctx / 1 slot]
|
||||
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model
|
||||
on Dankclaw (frank-default-large, active 2026-08-24)
|
||||
- [[eemin-rocmi4-mtp]] — the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model
|
||||
on Dankclaw (frank-default-large, active 2026-08-25)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the retired kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
|
||||
model on Dankclaw (superseded 2026-08-25 by the ROCmI4 embedded-MTP model)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||||
|
||||
@@ -15,10 +15,13 @@ timestamp: 2026-08-24
|
||||
|
||||
# kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
|
||||
|
||||
> **Active 2026-08-24.** `frank-default-large` serves this model. It
|
||||
> replaced the jcbtc v3 CIRU ActiveFPX PromptForge payload on the same
|
||||
> ROCmFPX runtime and the same CIRU llama.cpp binary. See
|
||||
> [[ciru-activefpx-promptforge]] for the retired v3 page.
|
||||
> **Retired 2026-08-25.** The kingjones ROCmFP4 STRIX MTP payload has been
|
||||
> replaced by the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on the same
|
||||
> port and iGPU, under a new runtime (charlie12345/ROCmFPX ROCmI4 W4A4). See
|
||||
> [[eemin-rocmi4-mtp]]. The old kingjones GGUF + standalone MTP Q4_0 draft are
|
||||
> retired on this slot. The CIRU ROCmFPX runtime/binary is also retired here.
|
||||
|
||||
> _This page documents the retired model and its runbook for reference._
|
||||
|
||||
`frank-default-large` (port 7770) serves
|
||||
`Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
|
||||
|
||||
@@ -96,6 +96,8 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[eemin-rocmi4-mtp]] — the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw (frank-default-large, active)
|
||||
- [[kingjones-strix-mtp]] — the RETIRED kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (superseded 2026-08-25)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active)
|
||||
|
||||
Reference in new issue
Block a user