serverAppConfigs/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw	ACTIVE frank-default-large runbook. kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.	llama.cpp rocm dankclaw gfx1151 kingjones rocmfp4 strix-halo qwen4exp	2026-08-30

kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw

Active again since 2026-08-30 on the model card's pinned runtime: the kingjones30/ROCmFPX fork (replaced the agentionai FAST deployment which ran on charlie's build-qwen38). Text-only tier; the agentionai f16 vision tower is paired for mmproj and loads clean. Speculative decoding is OFF (the card ships no spec flags; the Flash-Next MTP heads do not load on this loader family — see MTP Draft Head).

frank-default-large (port 7770) serves Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf (single file, 113.47 GiB) from /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/ via scripts/serve-frank-large.sh + scripts/frank-default-large.env and the Qwen-Sharp chat template.

Model Architecture

Base model: Qwen3.8-Flash-Next (177B parameters total).

Component	Params	Details
MoE experts	125.8B	512 experts, 10 used per token
Attention	dense	48 layers (12 attn + 36 SSM)
PLE table	51.2B	per_layer_token_embd, Q8_0
lm head	~0.3%	Q6_K

The PLE (per-layer embedding) n-gram table is the only thing that differs between STRIX, STRIX_LEAN, and FAST tiers. Everything else is byte-identical.

File Details

Field	Value
File	Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf
Size	113.47 GiB
Format	Single GGUF (not sharded)
file_type	105 (Q4_0_ROCMFP4_STRIX)
General	113.47 GiB, 5.51 bpw

Tensor composition:

Tensor group	Type
MoE expert weights (ffn_*_exps, 144)	TYPE_101 (ROCmFP4, 4.251 bpw)
Shared expert (ffn_*_shexp, 144)	TYPE_101
Attention (attn_*, 120)	half TYPE_100, half TYPE_101
PLE (per_layer_token_embd.weight)	Q8_0 — 50.66 GiB
token_embd.weight	Q6_K
output.weight (lm head)	Q6_K
Norms / biases	F32

File Type Reference

Tier	file_type	PLE table	Total size
FAST	TYPE_106	25.34 GiB	87.94 GiB
STRIX_LEAN	TYPE_105 (old STRIX_LEAN)	35.76 GiB	98.49 GiB
STRIX	105 (Q4_0_ROCMFP4_STRIX)	50.66 GiB	113.47 GiB

TYPE_100 = 4.506 bpw, TYPE_101 = 4.251 bpw (measured exactly by solving three files for two unknowns).

Memory

The PLE table never enters GPU VRAM. It streams from the file through the OS page cache.

Context	GTT at load	GTT after prompt	GPU resident
8,192	63.3 GiB	63.6 GiB	~63 GiB
32,768	64.1 GiB	64.8 GiB
65,536	65.1 GiB	66.2 GiB
131,072	67.2 GiB	69.1 GiB	~69 GiB

~63 GiB is the non-PLE mass plus compute buffers. The PLE table does not grow GPU memory regardless of context depth.

Three critical consequences:

1. --no-mmap gets you OOM-killed. It forces the PLE into anonymous memory (neither file-backed nor reclaimable). The cgroup killer takes the process with nothing in the server log — only dmesg shows the evidence.
2. Don't force PLE to CPU with -ot. Measured generation dropped from ~23 to 13.4 tok/s. The kernel streams it better than any offload hack.
3. A bigger tier costs disk, not GPU memory. If you have storage, take STRIX.

Performance

Measured on Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), 49/49 offload. Prompts sized to 85% of context window:

Context	Prompt tokens	pp tok/s	gen tok/s
8,192	6,963	279	22.02
32,768	27,852	264	20.52
65,536	55,705	234	18.23
131,072	111,411	185	15.34

Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s generation (median of 3 runs).

Benchmark: threads-batch tuning

Sweeped --threads-batch (16, 32, 48, 64, 80, 96, 128) with a 64K-token prompt via HTTP API (max_tokens=1). Fresh server start per value.

threads-batch	prefill tok/s	wall time
16	373.5	62.9s
32	354.4	66.3s
48	344.1	68.3s
64	344.7	68.2s
80	344.4	68.2s
96	357.1	65.8s
128	321.1	73.2s

Optimal: --threads-batch 16 (373.5 tok/s). Lower thread batching wins for prefill on this model -- fewer threads compete for GPU resources during the prompt processing phase. The default of 32 is 5.1% slower. Above 64, performance plateaus then declines (128 is 14% worse than optimal).

Benchmark: decode throughput

Sweeped --threads-batch (8, 16, 32) with a ~23.5K-token code prompt and 256 generated tokens via HTTP API. Fresh server start per value.

threads-batch	gen tokens	wall time	gen tok/s
16	256	74.8s	3.4
32	256	75.0s	3.4
8	256	76.6s	3.3

Decode throughput is essentially flat across all tested values. TB=16 and TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s. Combined with the 5.1% prefill advantage at lower thread counts, --threads-batch 16 is the optimal setting for both phases.

Runtime build (kingjones30/ROCmFPX)

The card pins its own fork: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the qwen4exp-on-rocmfpx-d3ca537 patch, so ROCmFP4 tensor types and the qwen4exp arch live in one tree. Built at /home/dank/GitPackages/kingjones-ROCmFPX/build from commit 36e9acd (2026-08-27 card revision) with the card's recipe:

git clone https://github.com/kingjones30/ROCmFPX.git
cd ROCmFPX
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF
cmake --build build --target llama-server llama-quantize -j16

• -DLLAMA_BUILD_WEBUI=OFF is required on this box: the tree (post-WebUI llama.cpp) provisions embedded UI assets via npm or the HF llama-ui bucket; npm is absent here and the bucket download half-failed, aborting the link at 98%. The API server is unaffected. All other compile work succeeded with zero errors, confirming the card's clean-clone claim.
• All 35 flags our serve script passes were verified present in this tree (--spec-type, --cache-idle-slots, --ctx-checkpoints, --reasoning-budget, --mmproj, …).

Serving Profile

Deployed args (scripts/frank-default-large.env — our validated profile, unchanged from the FAST deployment except the binary and model):

llama-server \
  --model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf \
  --host 0.0.0.0 --port 7770 --jinja \
  --device ROCm0 --gpu-layers 999 \
  --fit off --flash-attn on \
  --ctx-size 401408 --parallel 2 \
  --batch-size 4096 --ubatch-size 512 \
  --cache-type-k f16 --cache-type-v f16 \
  --threads 16 --threads-batch 16 \
  --ctx-checkpoints 16 --cache-ram 8192 \
  --cache-prompt --cache-idle-slots --metrics \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
  --presence-penalty 1.5 \
  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
  --chat-template-file chat-template.jinja \
  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
  --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf

Key choices:

• --fit off — auto-fit segfaults on this model's qwen4exp hyper-connection/indexer configuration.
• mmap (default) — do NOT use --no-mmap. See memory section.
• --flash-attn on works unpatched on this model — verified live 2026-08-30 (graph built, model served). The QSA F16 ggml_fill assert that forced agentionai-fast-imatrix onto the patched charlie tree did not fire for the kingjones STRIX file; if a future build aborts with GGML_ASSERT(a-type == GGML_TYPE_F32), port scripts/patches/qwen4exp-qsa-f32-mask.patch into this tree.
• --ctx-size 401408 --parallel 2 (200704/slot) — carried from the FAST deployment; the card's own ladder proves the window is nearly free (QSA caps KV: ~8 GiB total 8k→256k). Verified live at depth: a 111k-token prompt prefills at ~230 t/s while a second slot decodes.
• --ubatch-size 512 — card re-measurement: -ub 1024/2048/4096 show no effect; 512 is already right on this hardware.
• --mmproj — the STRIX repo ships no vision tower; the agentionai FAST package's f16 mmproj (same base model) is paired and loads (one benign warmup: CLIP graph uses unsupported operators warning).
• No spec flags — speculative decoding is off; see below.

MTP Draft Head — not loadable on this loader family

The previous draft of this runbook ran EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF (Q8_0, 4.1 GB) as --model-draft. It does not load: the qwen4exp loader requires a full per-layer blk.<il>.hc_attn_norm.weight set and every Flash-Next-era head carries only blk.48 (verified 3x against multiple mains on 2026-08-28/29, see agentionai-fast-imatrix known issues). The service therefore runs with no speculative decoding — the card's own Usage block ships none. The card's published +50% decode figure was measured with Aristo94/EngramHalo.cpp, a fork we do not build. Revisit only if a head with the complete hyper-connection norm set (or a loader that tolerates partial draft tensors) appears.

Validation (2026-08-30 swap)

1. Integrity: on-host file sha256sum = c6770d7442a06bf1d78edf28cec83e1ec93afdd34664c23ff898807b6b9349fa, byte-identical to the HF LFS oid. No re-download needed.
2. Load: systemd unit healthy; model loaded in ~225 s; health 200.
3. Generation: /v1/chat/completions returns correct reasoning_content + content through the Sharp template (deepseek format, preserve_reasoning).
4. Real load: an operator session's ~111k-token prompt prefilled at 213–232 t/s under the new runtime, matching the card's 131k rung (196 t/s), with checkpoints + QSA KV streaming live.

Deployment

Download

# on Dankclaw
TOKEN=$(cat ~/.cache/huggingface/token)
DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
mkdir -p "$DIR"
curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
  --retry 5 --retry-delay 3 --retry-max-time 7200 \
  -H "Authorization: Bearer $TOKEN" \
  "https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf"

The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.

Swap procedure (as run 2026-08-30)

1. Build the runtime (see Runtime build) BEFORE touching the service.
2. Point the model file's directory at the canonical path (a plain mv — never delete a 113 GiB payload to re-fetch it).
3. Edit scripts/frank-default-large.env (+ serve script defaults) via git: commit, push, ./scripts/deploy.sh dankclaw --service frank-default-large.
4. Verify health: curl http://192.168.0.143:7770/health and a real /v1/chat/completions probe.

Rollback: the FAST deployment is one env commit away (LLAMA_BIN, MODEL_DIR, MODEL_FILE, MMPROJ_PATH); the charlie build-qwen38 binary and the agentionai payload remain on disk.

Config files

• Env: scripts/frank-default-large.env
• Serve: scripts/serve-frank-large.sh
• Chat template: scripts/chat-template.jinja

Known Issues

1. Pinned fork required. Stock llama.cpp and plain charlie12345/ ROCmFPX each lack half the tree (qwen4exp arch / ROCmFP4 types). Use kingjones30/ROCmFPX, or the equivalent merged charlie tree (/home/dank/GitPackages/ROCmFPX, PR #27742 commits).
2. Single-file payload. Not sharded. Use hf download or a resumable curl. Browser download unreliable for 113 GiB.
3. Model load is slow. A few minutes cold for the 113 GiB file with a fresh KV cache. Health returns 503 while loading. Do not kill the process. deploy.sh waits up to 20 minutes.
4. Kernel spam about amdgpu: SVM mapping failed is a known documented symptom, not root cause. Investigate other causes before assuming memory pressure.

Citations

• kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF
• Qwen/Qwen3.8-Flash-Next base model
• kingjones30/ROCmFPX — the card's pinned fork (charlie12345/ROCmFPX + qwen4exp-on-rocmfpx-d3ca537.patch, mirrored in the HF repo)
• charlie12345/ROCmFPX — upstream fork defining the ROCmFP4 formats; our rollback tree with PR #27742 merged
• llama.cpp PR #27739 — MTP draft head support