docs: link STRIX runbook from top-level index
This commit is contained in:
Frank
@@ -0,0 +1,195 @@
|
||||
---
|
||||
type: reference
|
||||
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
|
||||
description: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the ROCmFPX llama.cpp fork. 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF. frank-default-large.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- kingjones
|
||||
- rocmfp4
|
||||
- strix-halo
|
||||
- qwen4exp
|
||||
timestamp: 2026-08-27
|
||||
---
|
||||
|
||||
# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
|
||||
|
||||
> **Active since 2026-08-27.** Replaced the STRIX_LEAN deployment on the same
|
||||
> build. Same GPU memory profile, larger PLE table.
|
||||
|
||||
`frank-default-large` (port 7770) serves
|
||||
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX` from `kingjones777` on Hugging Face.
|
||||
A single GGUF file, 113.47 GiB. Text-only (no mmproj, no MTP).
|
||||
|
||||
## Model Architecture
|
||||
|
||||
Base model: Qwen3.8-Flash-Next (177B parameters total).
|
||||
|
||||
| Component | Params | Details |
|
||||
|-----------|--------|---------|
|
||||
| MoE experts | 125.8B | 512 experts, 10 used per token |
|
||||
| Attention | dense | 48 layers (12 attn + 36 SSM) |
|
||||
| PLE table | 51.2B | `per_layer_token_embd`, Q8_0 |
|
||||
| lm head | ~0.3% | Q6_K |
|
||||
|
||||
The PLE (per-layer embedding) n-gram table is the only thing that differs
|
||||
between STRIX, STRIX_LEAN, and FAST tiers. Everything else is byte-identical.
|
||||
|
||||
## File Details
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| File | `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` |
|
||||
| Size | 113.47 GiB |
|
||||
| Format | Single GGUF (not sharded) |
|
||||
| file_type | 105 (`Q4_0_ROCMFP4_STRIX`) |
|
||||
| General | 113.47 GiB, 5.51 bpw |
|
||||
|
||||
Tensor composition:
|
||||
|
||||
| Tensor group | Type |
|
||||
|---|---|
|
||||
| MoE expert weights (`ffn_*_exps`, 144) | TYPE_101 (ROCmFP4, 4.251 bpw) |
|
||||
| Shared expert (`ffn_*_shexp`, 144) | TYPE_101 |
|
||||
| Attention (`attn_*`, 120) | half TYPE_100, half TYPE_101 |
|
||||
| PLE (`per_layer_token_embd.weight`) | Q8_0 — 50.66 GiB |
|
||||
| `token_embd.weight` | Q6_K |
|
||||
| `output.weight` (lm head) | Q6_K |
|
||||
| Norms / biases | F32 |
|
||||
|
||||
## File Type Reference
|
||||
|
||||
| Tier | file_type | PLE table | Total size |
|
||||
|------|-----------|-----------|------------|
|
||||
| FAST | TYPE_106 | 25.34 GiB | 87.94 GiB |
|
||||
| STRIX_LEAN | TYPE_105 (old STRIX_LEAN) | 35.76 GiB | 98.49 GiB |
|
||||
| **STRIX** | **105 (`Q4_0_ROCMFP4_STRIX`)** | **50.66 GiB** | **113.47 GiB** |
|
||||
|
||||
TYPE_100 = 4.506 bpw, TYPE_101 = 4.251 bpw (measured exactly by solving
|
||||
three files for two unknowns).
|
||||
|
||||
## Memory
|
||||
|
||||
The PLE table never enters GPU VRAM. It streams from the file through the OS
|
||||
page cache.
|
||||
|
||||
| Context | GTT at load | GTT after prompt | GPU resident |
|
||||
|---------|-------------|-------------------|-------------|
|
||||
| 8,192 | 63.3 GiB | 63.6 GiB | ~63 GiB |
|
||||
| 32,768 | 64.1 GiB | 64.8 GiB | |
|
||||
| 65,536 | 65.1 GiB | 66.2 GiB | |
|
||||
| 131,072 | 67.2 GiB | 69.1 GiB | ~69 GiB |
|
||||
|
||||
~63 GiB is the non-PLE mass plus compute buffers. **The PLE table does not
|
||||
grow GPU memory regardless of context depth.**
|
||||
|
||||
Three critical consequences:
|
||||
|
||||
1. **`--no-mmap` gets you OOM-killed.** It forces the PLE into anonymous memory
|
||||
(neither file-backed nor reclaimable). The cgroup killer takes the process
|
||||
with nothing in the server log — only `dmesg` shows the evidence.
|
||||
2. **Don't force PLE to CPU with `-ot`.** Measured generation dropped from
|
||||
~23 to 13.4 tok/s. The kernel streams it better than any offload hack.
|
||||
3. **A bigger tier costs disk, not GPU memory.** If you have storage, take
|
||||
STRIX.
|
||||
|
||||
## Performance
|
||||
|
||||
Measured on Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), 49/49 offload. Prompts
|
||||
sized to 85% of context window:
|
||||
|
||||
| Context | Prompt tokens | pp tok/s | gen tok/s |
|
||||
|---------|--------------|----------|-----------|
|
||||
| 8,192 | 6,963 | 279 | **22.02** |
|
||||
| 32,768 | 27,852 | 264 | **20.52** |
|
||||
| 65,536 | 55,705 | 234 | **18.23** |
|
||||
| 131,072 | 111,411 | 185 | **15.34** |
|
||||
|
||||
Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s
|
||||
generation (median of 3 runs).
|
||||
|
||||
## Serving Profile
|
||||
|
||||
Runtime: charlie12345/ROCmFPX `llama-server` built at
|
||||
`/home/dank/GitPackages/ROCmFPX/build-qwen38`.
|
||||
|
||||
Required flags:
|
||||
|
||||
```
|
||||
llama-server \
|
||||
--model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf \
|
||||
--host 0.0.0.0 --port 7770 --jinja \
|
||||
--device ROCm0 --gpu-layers 999 \
|
||||
--fit off --flash-attn on \
|
||||
--ctx-size 524288 --parallel 2 \
|
||||
--batch-size 4096 --ubatch-size 4096 \
|
||||
--cache-type-k f16 --cache-type-v f16 \
|
||||
--threads 16 --threads-batch 32 \
|
||||
--ctx-checkpoints 16 --cache-ram 8192 \
|
||||
--cache-prompt --cache-idle-slots --metrics \
|
||||
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
|
||||
--presence-penalty 1.5 \
|
||||
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
|
||||
--chat-template-file chat-template.jinja \
|
||||
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
|
||||
```
|
||||
|
||||
Key choices:
|
||||
|
||||
- **`--fit off`** — auto-fit code segfaults on this model's qwen4exp
|
||||
hyper-connection/indexer configuration.
|
||||
- **mmap (default)** — do NOT use `--no-mmap`. See memory section.
|
||||
- **`--gpu-layers 999`** — all layers on SVM-backed GPU pool.
|
||||
- **`--ctx-size 524288`** — above model's max (131072); llama.cpp clamps
|
||||
internally.
|
||||
|
||||
## Deployment
|
||||
|
||||
### Download
|
||||
|
||||
```bash
|
||||
# on Dankclaw
|
||||
TOKEN=$(cat ~/.cache/huggingface/token)
|
||||
DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
|
||||
mkdir -p "$DIR"
|
||||
curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
|
||||
--retry 5 --retry-delay 3 --retry-max-time 7200 \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
"https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf"
|
||||
```
|
||||
|
||||
The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
|
||||
|
||||
### Swap procedure
|
||||
|
||||
1. Stop service: `sudo systemctl stop frank-default-large`
|
||||
2. Remove old model dir: `rm -rf /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN`
|
||||
3. Start service: `sudo systemctl start frank-default-large`
|
||||
4. Verify health: `curl http://192.168.0.143:7770/health`
|
||||
|
||||
### Config files
|
||||
|
||||
- Env: `scripts/frank-default-large.env`
|
||||
- Serve: `scripts/serve-frank-large.sh`
|
||||
- Chat template: `scripts/chat-template.jinja`
|
||||
|
||||
## Known Issues
|
||||
|
||||
1. **Patched llama.cpp required.** PR #27742 merged into the ROCmFPX fork.
|
||||
Stock builds reject these files (TYPE_100/101 have no number in stock ggml).
|
||||
2. **Single-file payload.** Not sharded. Use `hf download` or a resumable
|
||||
curl. Browser download unreliable for 113 GiB.
|
||||
3. **Model load is slow.** A few minutes cold for the 113 GiB file with a
|
||||
fresh KV cache. Health returns 503 while loading. Do not kill the process.
|
||||
`deploy.sh` waits up to 20 minutes.
|
||||
4. **Kernel spam about `amdgpu: SVM mapping failed`** is a known documented
|
||||
symptom, not root cause. Investigate other causes before assuming memory
|
||||
pressure.
|
||||
|
||||
## Citations
|
||||
|
||||
- [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
|
||||
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
|
||||
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture)
|
||||
@@ -108,6 +108,8 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[freetoken-frank-default-small]] — the active frank-default-small runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver (2026-09-02)
|
||||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (restored 2026-09-01)
|
||||
- [[kingjones-strix-rocmfp4]] — the active frank-default-large runbook:
|
||||
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B params, 113.47 GiB)
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
|
||||
Reference in New Issue
Block a user