docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
This commit is contained in:
Frank committed
1 parent
6d4a96a423
commit
5829a0bfe8
140 files changed
+1820
-86
No files matched your search
@@ -0,0 +1,186 @@
|
||||
---
|
||||
type: reference
|
||||
title: caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
|
||||
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-default-large.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- rocmi4
|
||||
- mtp
|
||||
- iu4
|
||||
timestamp: 2026-08-25
|
||||
---
|
||||
|
||||
# caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
|
||||
|
||||
> **Active 2026-08-25.** `frank-default-large` serves this model. It replaced
|
||||
> the kingjones777 ROCmFP4 STRIX MTP payload on the same port and the same
|
||||
> Strix Halo iGPU, running the new charlie12345/ROCmFPX ROCmI4 W4A4 build. See
|
||||
> [[kingjones-strix-mtp]] for the retired page.
|
||||
|
||||
`frank-default-large` (port 7770) serves
|
||||
`cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
|
||||
`eemin` namespace) from the **charlie12345/ROCmFPX** llama.cpp fork built with
|
||||
the opt-in `ROCmI4 W4A4` path. The model is `Q4_0_ROCMI4`: signed four-bit
|
||||
weights packed two per byte, quantized from the BF16 Qwen3.8-27B and carrying
|
||||
its **embedded NextN/MTP head** in the main GGUF — so there is no separate
|
||||
draft file. The native gfx1151 `v_wmma_i32_16x16x16_iu4` matrix instruction
|
||||
batch-verifies the MTP proposals for higher decode throughput.
|
||||
|
||||
## What This Is
|
||||
|
||||
**ROCmI4** is the GGUF weight format (signed 4-bit codes). **IU4** is the AMD
|
||||
WMMA instruction spelling that consumes them. **W4A4** means four-bit weights
|
||||
and four-bit activations during the accelerated matrix multiply. Ordinary
|
||||
one-token decode stays memory-bandwidth-bound (MMVQ); batched prompt processing
|
||||
and batched MTP target verification use the IU4 MMQ path. W4A4 is a lossy
|
||||
activation-quant speed path — treat it as a speed/quality tradeoff.
|
||||
|
||||
The model is a conversion/quantization of `Qwen/Qwen3.8-27B` (Unsloth lineage,
|
||||
Apache-2.0), not a retrained model. It reports architecture `qwen35`
|
||||
(M-RoPE), ~27.32B params, native 262144 context, 866 tensors, ftype 108.
|
||||
|
||||
## Model Files
|
||||
|
||||
| File | Size | SHA-256 |
|
||||
|------|------|---------|
|
||||
| `Qwen3.8-27B-Q4_0_ROCMI4.gguf` | 13.50 GiB (14,534,384,384 B) | `ad51ecbee55930ababc629c377943a7bbe50f8aafcf32340c7cdae6fa6b7bb3c` |
|
||||
| `mmproj-BF16.gguf` (vision, optional) | 0.87 GiB (931,146,432 B) | `83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53` |
|
||||
|
||||
Store under `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/`. The vision
|
||||
projector is optional for text-only use.
|
||||
|
||||
## Runtime Build
|
||||
|
||||
The stock llama.cpp build does not understand `Q4_0_ROCMI4` or the qualified
|
||||
IU4/MTP path. Use charlie12345/ROCmFPX at the pinned commit, built HIP-only
|
||||
(so the combined HIP/Vulkan scheduler cannot route the ROCmI4 tensors to
|
||||
Vulkan, which has no shader for this type) with the W4A4 option on.
|
||||
|
||||
| Artifact | Path |
|
||||
|----------|------|
|
||||
| Fork + pinned commit | `/home/dank/rocmi4-qwen38/ROCmFPX` @ `c49ebdbd5c9f01ec242369f9e7f7967855f80cba` |
|
||||
| Build | `cmake -S . -B build-rocmi4-w4a4 -G Ninja -DGGML_HIP=ON -DGGML_VULKAN=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_HIP_ROCMI4_W4A4=ON -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF` |
|
||||
| Binary | `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (version 257, `c49ebdb`) |
|
||||
| ROCm | system `/opt/rocm-7.2.4`, amdclang 22 |
|
||||
|
||||
Build a separate directory so the exact-int8 path (the default branch) stays
|
||||
available. Rollback: disable `-DGGML_HIP_ROCMI4_W4A4=ON` and rebuild — no model
|
||||
conversion or data change needed. The build prints
|
||||
`ROCmI4 W4A4: enabled for device 0` when the accelerated path is active.
|
||||
|
||||
## Model Download
|
||||
|
||||
Downloaded from the public
|
||||
`eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF` repo (the card's download section still
|
||||
names the still-private `cafonz/…`; use `eemin`). Verify against the repo's
|
||||
`checksums.sha256` after download:
|
||||
|
||||
```
|
||||
cd /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP
|
||||
sha256sum -c checksums.sha256
|
||||
```
|
||||
|
||||
## Launch Profile
|
||||
|
||||
The qualified MTP profile (from the model card and PR #94, verified on the
|
||||
live service): strict Qwen MTP, backend sampling, `n_max 4`, `p_min 0.60`.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
|
||||
B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
|
||||
C --> D[strict Qwen MTP<br/>n_max 4 / p_min 0.60<br/>backend sampling]
|
||||
D --> E[Sharp chat template<br/>reasoning deepseek]
|
||||
```
|
||||
|
||||
Key flags (see `scripts/serve-frank-large.sh`):
|
||||
|
||||
```
|
||||
--spec-type draft-mtp --spec-mtp-strict-qwen
|
||||
--spec-draft-device ROCm0 --spec-draft-ngl all
|
||||
--spec-draft-type-k f16 --spec-draft-type-v f16
|
||||
--spec-draft-n-max 4 --spec-draft-n-min 0
|
||||
--spec-draft-p-min 0.60 --spec-draft-backend-sampling
|
||||
```
|
||||
|
||||
Because the MTP head is **embedded**, no `--model-draft` is passed and the
|
||||
draft uses all layers on ROCm0. `--spec-mtp-strict-qwen` preserves the target
|
||||
path's greedy decisions; it does not remove the W4A4 activation approximation.
|
||||
|
||||
### Serving settings
|
||||
|
||||
- 65536 context (192K below the native 262144 maximum), 1 slot, f16 KV,
|
||||
context checkpoints, prompt cache. The lower limit leaves headroom for the
|
||||
model plus KV cache in the host's unified-memory budget. Concurrent requests
|
||||
queue; the server never preempts a running generation.
|
||||
- `-t 16 -tb 32`, batch 512 / ubatch 256 (the qualified W4A4 profile).
|
||||
- `--temp 0.7 --top-p 0.95`, reasoning on (`deepseek` puts `<think>` into
|
||||
`message.reasoning_content`).
|
||||
- Qwen-Sharp chat template (`--chat-template-file`), `enable_thinking` +
|
||||
`preserve_thinking` on, preserving the peg-native + deepseek shape the Pi
|
||||
agent expects.
|
||||
- Strix Halo unified memory + native gfx1151 detection: `HSA_OVERRIDE_GFX_VERSION=11.5.1`,
|
||||
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`.
|
||||
|
||||
### Service
|
||||
|
||||
- Unit: `frank-default-large.service`, port 7770
|
||||
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
|
||||
|
||||
## Tuning MTP
|
||||
|
||||
MTP gains are content-dependent: predictable code, structured data, and lists
|
||||
accept longer drafts than creative prose. The active conservative profile is
|
||||
`n4 / p0.60`; the previously qualified starting point was `n16 / p0.60`.
|
||||
|
||||
## ubatch Benchmark
|
||||
|
||||
Decode throughput depends on the ubatch (per-eval token batch). Run the sweep
|
||||
on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch:
|
||||
|
||||
```
|
||||
sudo systemctl stop frank-default-large
|
||||
./scripts/bench-ubatch.sh
|
||||
sudo systemctl start frank-default-large
|
||||
```
|
||||
|
||||
The script stops `frank-default-large` for a clean GPU, boots one server
|
||||
instance per ubatch on a scratch port, reads authoritative `print_timing` tok/s,
|
||||
and restarts the service. See [[frank-default-large-benchmark]].
|
||||
|
||||
## Known Issues
|
||||
|
||||
1. **gfx1151 only** for the W4A4 path. Other GPUs fall back to exact int8 MMQ.
|
||||
2. **Memory pressure:** the model is 13.5 GiB. Under `--no-mmap` with full
|
||||
offload it needs the whole tensor set at once; leave room in the 64 GiB
|
||||
unified pool (rocm-smi figure). Do not run a bench with the live service up
|
||||
or it OOMs at load.
|
||||
3. **`Q4_0_ROCMI4` + MTP flags need the ROCmFPX fork**; stock llama.cpp may
|
||||
reject the file or drop the optimized path.
|
||||
4. **W4A4 is lossy** (+~5.4% perplexity vs exact int8). Opt-in speed path.
|
||||
|
||||
## Measured Results (qualification)
|
||||
|
||||
From the model card, gfx1151, pinned build, W4A4 + strict MTP-16:
|
||||
|
||||
| Result | Value |
|
||||
|--------|-------|
|
||||
| HumanEval pilot, W4A4 IU4 (mean) | 49.40 tok/s |
|
||||
| Full HumanEval, W4A4 (mean / median) | 44.39 / 45.23 tok/s |
|
||||
| HumanEval / HumanEval+ pass@1 | 94.5% / 91.5% |
|
||||
| Non-speculative tg128 | ~13.8 tok/s |
|
||||
|
||||
Local smoke test 2026-08-25 on the iGPU: model loaded, strict-MTP active,
|
||||
draft acceptance 0.74 (mean 7.67), generation ~25 tok/s at a small output.
|
||||
The ubatch sweep above gives the production ubatch choice.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[kingjones-strix-mtp]] — the retired ROCmFP4 predecessor on this slot
|
||||
- [[ciru-activefpx-promptforge]] — the retired v3 CIRU PromptForge page
|
||||
- [[Dankclaw System Specs]] — the box
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark history
|
||||
Reference in new issue
Block a user