serverAppConfigs/docs/LLM/kingjones-strix-rocmfp4.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw	kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the ROCmFPX llama.cpp fork. 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF. frank-default-large.	llama.cpp rocm dankclaw gfx1151 kingjones rocmfp4 strix-halo qwen4exp	2026-08-27

kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw

Active since 2026-08-27. Replaced the STRIX_LEAN deployment on the same build. Same GPU memory profile, larger PLE table.

frank-default-large (port 7770) serves Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX from kingjones777 on Hugging Face. A single GGUF file, 113.47 GiB. Text-only (no mmproj). MTP draft head loaded as a separate GGUF sidecar (see MTP Draft Head below).

Model Architecture

Base model: Qwen3.8-Flash-Next (177B parameters total).

Component	Params	Details
MoE experts	125.8B	512 experts, 10 used per token
Attention	dense	48 layers (12 attn + 36 SSM)
PLE table	51.2B	per_layer_token_embd, Q8_0
lm head	~0.3%	Q6_K

The PLE (per-layer embedding) n-gram table is the only thing that differs between STRIX, STRIX_LEAN, and FAST tiers. Everything else is byte-identical.

File Details

Field	Value
File	Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf
Size	113.47 GiB
Format	Single GGUF (not sharded)
file_type	105 (Q4_0_ROCMFP4_STRIX)
General	113.47 GiB, 5.51 bpw

Tensor composition:

Tensor group	Type
MoE expert weights (ffn_*_exps, 144)	TYPE_101 (ROCmFP4, 4.251 bpw)
Shared expert (ffn_*_shexp, 144)	TYPE_101
Attention (attn_*, 120)	half TYPE_100, half TYPE_101
PLE (per_layer_token_embd.weight)	Q8_0 — 50.66 GiB
token_embd.weight	Q6_K
output.weight (lm head)	Q6_K
Norms / biases	F32

File Type Reference

Tier	file_type	PLE table	Total size
FAST	TYPE_106	25.34 GiB	87.94 GiB
STRIX_LEAN	TYPE_105 (old STRIX_LEAN)	35.76 GiB	98.49 GiB
STRIX	105 (Q4_0_ROCMFP4_STRIX)	50.66 GiB	113.47 GiB

TYPE_100 = 4.506 bpw, TYPE_101 = 4.251 bpw (measured exactly by solving three files for two unknowns).

Memory

The PLE table never enters GPU VRAM. It streams from the file through the OS page cache.

Context	GTT at load	GTT after prompt	GPU resident
8,192	63.3 GiB	63.6 GiB	~63 GiB
32,768	64.1 GiB	64.8 GiB
65,536	65.1 GiB	66.2 GiB
131,072	67.2 GiB	69.1 GiB	~69 GiB

~63 GiB is the non-PLE mass plus compute buffers. The PLE table does not grow GPU memory regardless of context depth.

Three critical consequences:

1. --no-mmap gets you OOM-killed. It forces the PLE into anonymous memory (neither file-backed nor reclaimable). The cgroup killer takes the process with nothing in the server log — only dmesg shows the evidence.
2. Don't force PLE to CPU with -ot. Measured generation dropped from ~23 to 13.4 tok/s. The kernel streams it better than any offload hack.
3. A bigger tier costs disk, not GPU memory. If you have storage, take STRIX.

Performance

Measured on Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), 49/49 offload. Prompts sized to 85% of context window:

Context	Prompt tokens	pp tok/s	gen tok/s
8,192	6,963	279	22.02
32,768	27,852	264	20.52
65,536	55,705	234	18.23
131,072	111,411	185	15.34

Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s generation (median of 3 runs).

Benchmark: threads-batch tuning

Sweeped --threads-batch (16, 32, 48, 64, 80, 96, 128) with a 64K-token prompt via HTTP API (max_tokens=1). Fresh server start per value.

threads-batch	prefill tok/s	wall time
16	373.5	62.9s
32	354.4	66.3s
48	344.1	68.3s
64	344.7	68.2s
80	344.4	68.2s
96	357.1	65.8s
128	321.1	73.2s

Optimal: --threads-batch 16 (373.5 tok/s). Lower thread batching wins for prefill on this model -- fewer threads compete for GPU resources during the prompt processing phase. The default of 32 is 5.1% slower. Above 64, performance plateaus then declines (128 is 14% worse than optimal).

Benchmark: decode throughput

Sweeped --threads-batch (8, 16, 32) with a ~23.5K-token code prompt and 256 generated tokens via HTTP API. Fresh server start per value.

threads-batch	gen tokens	wall time	gen tok/s
16	256	74.8s	3.4
32	256	75.0s	3.4
8	256	76.6s	3.3

Decode throughput is essentially flat across all tested values. TB=16 and TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s. Combined with the 5.1% prefill advantage at lower thread counts, --threads-batch 16 is the optimal setting for both phases.

Serving Profile

Runtime: charlie12345/ROCmFPX llama-server built at /home/dank/GitPackages/ROCmFPX/build-qwen38.

Required flags:

llama-server \
  --model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf \
  --host 0.0.0.0 --port 7770 --jinja \
  --device ROCm0 --gpu-layers 999 \
  --fit off --flash-attn on \
  --ctx-size 524288 --parallel 2 \
  --batch-size 4096 --ubatch-size 4096 \
  --cache-type-k f16 --cache-type-v f16 \
  --threads 16 --threads-batch 16 \
  --ctx-checkpoints 16 --cache-ram 8192 \
  --cache-prompt --cache-idle-slots --metrics \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
  --presence-penalty 1.5 \
  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
  --chat-template-file chat-template.jinja \
  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
  --spec-type draft-mtp,ngram-mod \
  --model-draft mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --spec-draft-n-max 4 --spec-draft-p-min 0.75 \
  --spec-draft-ngl 999 --spec-draft-device ROCm0 \
  --spec-draft-type-k f16 --spec-draft-type-v f16

Key choices:

• --fit off — auto-fit code segfaults on this model's qwen4exp hyper-connection/indexer configuration.
• mmap (default) — do NOT use --no-mmap. See memory section.
• --gpu-layers 999 — all layers on SVM-backed GPU pool.
• --ctx-size 524288 — above model's max (131072); llama.cpp clamps internally.
• --spec-type draft-mtp,ngram-mod — MTP draft head + ngram-mod fallback. The ngram-mod combo is recommended by the HF card; the prior dense 27B measurement showed ngram under MTP hurt by 4-5 t/s (kingjones-strix-mtp). Benchmark both before committing.
• --spec-draft-p-min 0.75 — confidence gate. Without it, low-acceptance prose loses more to rejected drafts than it gains (per the HF card).

MTP Draft Head

The MTP (multi-token-prediction) draft head from EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF (Q8_0, 4.1 GB). Architecture qwen4exp, matching the main model. Loaded as a separate GGUF sidecar via --model-draft.

Field	Value
File	mtp-Qwen3.8-Flash-Next-Q8_0.gguf
Size	4.1 GB (4,137,429,088 B)
Quant	Q8_0
Architecture	qwen4exp
Source	EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF

Measured (HF card, gfx1151, UD-IQ3_XXS target, temp 0): decode +50% on code (23.5 → 35.7 t/s, ~85% acceptance), +49% at 156K context depth. Speculative decoding is lossless at temp 0. Our service runs at temp 0.7, so expect lower acceptance; the p_min 0.75 gate compensates.

Caveats:

• The MTP head is from the original Qwen/Qwen3.8-Flash-Next; our model is the "Uncensored" (abliterated) variant. Abliteration modifies a small number of weight vectors, not the full model — acceptance may be marginally lower.
• The HF card's tested fork is Aristo94/EngramHalo.cpp, not our charlie12345/ROCmFPX. Both have PRs #27739 (MTP) + #27742 (qwen4exp) merged, so the MTP mechanism is the same upstream code. If the draft load crashes, check for Strix-Halo kernel patch differences first.
• Set SPEC_TYPE=none in scripts/frank-default-large.env to disable.

Deployment

Download

# on Dankclaw
TOKEN=$(cat ~/.cache/huggingface/token)
DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
mkdir -p "$DIR"
curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
  --retry 5 --retry-delay 3 --retry-max-time 7200 \
  -H "Authorization: Bearer $TOKEN" \
  "https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf"

The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.

Swap procedure

1. Stop service: sudo systemctl stop frank-default-large
2. Remove old model dir: rm -rf /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN
3. Start service: sudo systemctl start frank-default-large
4. Verify health: curl http://192.168.0.143:7770/health

Config files

• Env: scripts/frank-default-large.env
• Serve: scripts/serve-frank-large.sh
• Chat template: scripts/chat-template.jinja

Known Issues

1. Patched llama.cpp required. PR #27742 merged into the ROCmFPX fork. Stock builds reject these files (TYPE_100/101 have no number in stock ggml).
2. Single-file payload. Not sharded. Use hf download or a resumable curl. Browser download unreliable for 113 GiB.
3. Model load is slow. A few minutes cold for the 113 GiB file with a fresh KV cache. Health returns 503 while loading. Do not kill the process. deploy.sh waits up to 20 minutes.
4. Kernel spam about amdgpu: SVM mapping failed is a known documented symptom, not root cause. Investigate other causes before assuming memory pressure.

Citations

• kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF
• Qwen/Qwen3.8-Flash-Next base model
• charlie12345/ROCmFPX — fork with PR #27742 (qwen4exp architecture)
• EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF — MTP draft head sidecar
• llama.cpp PR #27739 — MTP draft head support