Files

T

Frank 8212d4451a docs: rename frank-default-large/small/sightsound to frank-large/small/npu across vault

Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.

2026-09-03 16:17:14 -07:00

7.7 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw

Active 2026-08-25. frank-large serves this model. It replaced the kingjones777 ROCmFP4 STRIX MTP payload on the same port and the same Strix Halo iGPU, running the new charlie12345/ROCmFPX ROCmI4 W4A4 build. See kingjones-strix-mtp for the retired page.

frank-large (port 7770) serves cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF (published under the public eemin namespace) from the charlie12345/ROCmFPX llama.cpp fork built with the opt-in ROCmI4 W4A4 path. The model is Q4_0_ROCMI4: signed four-bit weights packed two per byte, quantized from the BF16 Qwen3.8-27B and carrying its embedded NextN/MTP head in the main GGUF — so there is no separate draft file. The native gfx1151 v_wmma_i32_16x16x16_iu4 matrix instruction batch-verifies the MTP proposals for higher decode throughput.

What This Is

ROCmI4 is the GGUF weight format (signed 4-bit codes). IU4 is the AMD WMMA instruction spelling that consumes them. W4A4 means four-bit weights and four-bit activations during the accelerated matrix multiply. Ordinary one-token decode stays memory-bandwidth-bound (MMVQ); batched prompt processing and batched MTP target verification use the IU4 MMQ path. W4A4 is a lossy activation-quant speed path — treat it as a speed/quality tradeoff.

The model is a conversion/quantization of Qwen/Qwen3.8-27B (Unsloth lineage, Apache-2.0), not a retrained model. It reports architecture qwen35 (M-RoPE), ~27.32B params, native 262144 context, 866 tensors, ftype 108.

Model Files

File	Size	SHA-256
`Qwen3.8-27B-Q4_0_ROCMI4.gguf`	13.50 GiB (14,534,384,384 B)	`ad51ecbee55930ababc629c377943a7bbe50f8aafcf32340c7cdae6fa6b7bb3c`
`mmproj-BF16.gguf` (vision, optional)	0.87 GiB (931,146,432 B)	`83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53`

Store under /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/. The vision projector is optional for text-only use.

Runtime Build

The stock llama.cpp build does not understand Q4_0_ROCMI4 or the qualified IU4/MTP path. Use charlie12345/ROCmFPX at the pinned commit, built HIP-only (so the combined HIP/Vulkan scheduler cannot route the ROCmI4 tensors to Vulkan, which has no shader for this type) with the W4A4 option on.

Artifact	Path
Fork + pinned commit	`/home/dank/rocmi4-qwen38/ROCmFPX` @ `c49ebdbd5c9f01ec242369f9e7f7967855f80cba`
Build	`cmake -S . -B build-rocmi4-w4a4 -G Ninja -DGGML_HIP=ON -DGGML_VULKAN=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_HIP_ROCMI4_W4A4=ON -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF`
Binary	`/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (version 257, `c49ebdb`)
ROCm	system `/opt/rocm-7.2.4`, amdclang 22

Build a separate directory so the exact-int8 path (the default branch) stays available. Rollback: disable -DGGML_HIP_ROCMI4_W4A4=ON and rebuild — no model conversion or data change needed. The build prints ROCmI4 W4A4: enabled for device 0 when the accelerated path is active.

Model Download

Downloaded from the public eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF repo (the card's download section still names the still-private cafonz/…; use eemin). Verify against the repo's checksums.sha256 after download:

cd /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP
sha256sum -c checksums.sha256

Launch Profile

The qualified MTP profile (from the model card and PR #94, verified on the live service): strict Qwen MTP, backend sampling, n_max 4, p_min 0.60.

graph TD
    A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
    B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
    C --> D[strict Qwen MTP<br/>n_max 4 / p_min 0.60<br/>backend sampling]
    D --> E[Sharp chat template<br/>reasoning deepseek]

Key flags (see scripts/serve-frank-large.sh):

--spec-type draft-mtp --spec-mtp-strict-qwen
--spec-draft-device ROCm0 --spec-draft-ngl all
--spec-draft-type-k f16 --spec-draft-type-v f16
--spec-draft-n-max 4 --spec-draft-n-min 0
--spec-draft-p-min 0.60 --spec-draft-backend-sampling

Because the MTP head is embedded, no --model-draft is passed and the draft uses all layers on ROCm0. --spec-mtp-strict-qwen preserves the target path's greedy decisions; it does not remove the W4A4 activation approximation.

Serving settings

65536 context (192K below the native 262144 maximum), 1 slot, f16 KV, context checkpoints, prompt cache. The lower limit leaves headroom for the model plus KV cache in the host's unified-memory budget. Concurrent requests queue; the server never preempts a running generation.
-t 16 -tb 32, batch 512 / ubatch 256 (the qualified W4A4 profile).
--temp 0.7 --top-p 0.95, reasoning on (deepseek puts <think> into message.reasoning_content).
Qwen-Sharp chat template (--chat-template-file), enable_thinking + preserve_thinking on, preserving the peg-native + deepseek shape the Pi agent expects.
Strix Halo unified memory + native gfx1151 detection: HSA_OVERRIDE_GFX_VERSION=11.5.1, GGML_HIP_ENABLE_UNIFIED_MEMORY=1.

Service

Unit: frank-large.service, port 7770
Script: scripts/serve-frank-large.sh, env scripts/frank-large.env

Tuning MTP

MTP gains are content-dependent: predictable code, structured data, and lists accept longer drafts than creative prose. The active conservative profile is n4 / p0.60; the previously qualified starting point was n16 / p0.60.

ubatch Benchmark

Decode throughput depends on the ubatch (per-eval token batch). Run the sweep on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch:

sudo systemctl stop frank-large
./scripts/bench-ubatch.sh
sudo systemctl start frank-large

The script stops frank-large for a clean GPU, boots one server instance per ubatch on a scratch port, reads authoritative print_timing tok/s, and restarts the service. See frank-large-benchmark.

Known Issues

gfx1151 only for the W4A4 path. Other GPUs fall back to exact int8 MMQ.
Memory pressure: the model is 13.5 GiB. Under --no-mmap with full offload it needs the whole tensor set at once; leave room in the 64 GiB unified pool (rocm-smi figure). Do not run a bench with the live service up or it OOMs at load.
Q4_0_ROCMI4 + MTP flags need the ROCmFPX fork; stock llama.cpp may reject the file or drop the optimized path.
W4A4 is lossy (+~5.4% perplexity vs exact int8). Opt-in speed path.

Measured Results (qualification)

From the model card, gfx1151, pinned build, W4A4 + strict MTP-16:

Result	Value
HumanEval pilot, W4A4 IU4 (mean)	49.40 tok/s
Full HumanEval, W4A4 (mean / median)	44.39 / 45.23 tok/s
HumanEval / HumanEval+ pass@1	94.5% / 91.5%
Non-speculative tg128	~13.8 tok/s

Local smoke test 2026-08-25 on the iGPU: model loaded, strict-MTP active, draft acceptance 0.74 (mean 7.67), generation ~25 tok/s at a small output. The ubatch sweep above gives the production ubatch choice.

7.7 KiB Raw Blame History