docs: link STRIX runbook from top-level index

2026-08-27 03:01:27 -07:00
parent 0cf1f02bb2
commit 263fa99c16
4 changed files with 225 additions and 24 deletions
@@ -0,0 +1,195 @@
+---
+type: reference
+title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
+description: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the ROCmFPX llama.cpp fork. 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF. frank-default-large.
+tags:
+  - llama.cpp
+  - rocm
+  - dankclaw
+  - gfx1151
+  - kingjones
+  - rocmfp4
+  - strix-halo
+  - qwen4exp
+timestamp: 2026-08-27
+---
+
+# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
+
+> **Active since 2026-08-27.** Replaced the STRIX_LEAN deployment on the same
+> build. Same GPU memory profile, larger PLE table.
+
+`frank-default-large` (port 7770) serves
+`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX` from `kingjones777` on Hugging Face.
+A single GGUF file, 113.47 GiB. Text-only (no mmproj, no MTP).
+
+## Model Architecture
+
+Base model: Qwen3.8-Flash-Next (177B parameters total).
+
+| Component | Params | Details |
+|-----------|--------|---------|
+| MoE experts | 125.8B | 512 experts, 10 used per token |
+| Attention | dense | 48 layers (12 attn + 36 SSM) |
+| PLE table | 51.2B | `per_layer_token_embd`, Q8_0 |
+| lm head | ~0.3% | Q6_K |
+
+The PLE (per-layer embedding) n-gram table is the only thing that differs
+between STRIX, STRIX_LEAN, and FAST tiers. Everything else is byte-identical.
+
+## File Details
+
+| Field | Value |
+|-------|-------|
+| File | `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` |
+| Size | 113.47 GiB |
+| Format | Single GGUF (not sharded) |
+| file_type | 105 (`Q4_0_ROCMFP4_STRIX`) |
+| General | 113.47 GiB, 5.51 bpw |
+
+Tensor composition:
+
+| Tensor group | Type |
+|---|---|
+| MoE expert weights (`ffn_*_exps`, 144) | TYPE_101 (ROCmFP4, 4.251 bpw) |
+| Shared expert (`ffn_*_shexp`, 144) | TYPE_101 |
+| Attention (`attn_*`, 120) | half TYPE_100, half TYPE_101 |
+| PLE (`per_layer_token_embd.weight`) | Q8_0 — 50.66 GiB |
+| `token_embd.weight` | Q6_K |
+| `output.weight` (lm head) | Q6_K |
+| Norms / biases | F32 |
+
+## File Type Reference
+
+| Tier | file_type | PLE table | Total size |
+|------|-----------|-----------|------------|
+| FAST | TYPE_106 | 25.34 GiB | 87.94 GiB |
+| STRIX_LEAN | TYPE_105 (old STRIX_LEAN) | 35.76 GiB | 98.49 GiB |
+| **STRIX** | **105 (`Q4_0_ROCMFP4_STRIX`)** | **50.66 GiB** | **113.47 GiB** |
+
+TYPE_100 = 4.506 bpw, TYPE_101 = 4.251 bpw (measured exactly by solving
+three files for two unknowns).
+
+## Memory
+
+The PLE table never enters GPU VRAM. It streams from the file through the OS
+page cache.
+
+| Context | GTT at load | GTT after prompt | GPU resident |
+|---------|-------------|-------------------|-------------|
+| 8,192 | 63.3 GiB | 63.6 GiB | ~63 GiB |
+| 32,768 | 64.1 GiB | 64.8 GiB | |
+| 65,536 | 65.1 GiB | 66.2 GiB | |
+| 131,072 | 67.2 GiB | 69.1 GiB | ~69 GiB |
+
+~63 GiB is the non-PLE mass plus compute buffers. **The PLE table does not
+grow GPU memory regardless of context depth.**
+
+Three critical consequences:
+
+1. **`--no-mmap` gets you OOM-killed.** It forces the PLE into anonymous memory
+   (neither file-backed nor reclaimable). The cgroup killer takes the process
+   with nothing in the server log — only `dmesg` shows the evidence.
+2. **Don't force PLE to CPU with `-ot`.** Measured generation dropped from
+   ~23 to 13.4 tok/s. The kernel streams it better than any offload hack.
+3. **A bigger tier costs disk, not GPU memory.** If you have storage, take
+   STRIX.
+
+## Performance
+
+Measured on Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), 49/49 offload. Prompts
+sized to 85% of context window:
+
+| Context | Prompt tokens | pp tok/s | gen tok/s |
+|---------|--------------|----------|-----------|
+| 8,192 | 6,963 | 279 | **22.02** |
+| 32,768 | 27,852 | 264 | **20.52** |
+| 65,536 | 55,705 | 234 | **18.23** |
+| 131,072 | 111,411 | 185 | **15.34** |
+
+Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s
+generation (median of 3 runs).
+
+## Serving Profile
+
+Runtime: charlie12345/ROCmFPX `llama-server` built at
+`/home/dank/GitPackages/ROCmFPX/build-qwen38`.
+
+Required flags:
+
+```
+llama-server \
+  --model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf \
+  --host 0.0.0.0 --port 7770 --jinja \
+  --device ROCm0 --gpu-layers 999 \
+  --fit off --flash-attn on \
+  --ctx-size 524288 --parallel 2 \
+  --batch-size 4096 --ubatch-size 4096 \
+  --cache-type-k f16 --cache-type-v f16 \
+  --threads 16 --threads-batch 32 \
+  --ctx-checkpoints 16 --cache-ram 8192 \
+  --cache-prompt --cache-idle-slots --metrics \
+  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
+  --presence-penalty 1.5 \
+  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
+  --chat-template-file chat-template.jinja \
+  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
+```
+
+Key choices:
+
+- **`--fit off`** — auto-fit code segfaults on this model's qwen4exp
+  hyper-connection/indexer configuration.
+- **mmap (default)** — do NOT use `--no-mmap`. See memory section.
+- **`--gpu-layers 999`** — all layers on SVM-backed GPU pool.
+- **`--ctx-size 524288`** — above model's max (131072); llama.cpp clamps
+  internally.
+
+## Deployment
+
+### Download
+
+```bash
+# on Dankclaw
+TOKEN=$(cat ~/.cache/huggingface/token)
+DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
+mkdir -p "$DIR"
+curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
+  --retry 5 --retry-delay 3 --retry-max-time 7200 \
+  -H "Authorization: Bearer $TOKEN" \
+  "https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf"
+```
+
+The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
+
+### Swap procedure
+
+1. Stop service: `sudo systemctl stop frank-default-large`
+2. Remove old model dir: `rm -rf /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN`
+3. Start service: `sudo systemctl start frank-default-large`
+4. Verify health: `curl http://192.168.0.143:7770/health`
+
+### Config files
+
+- Env: `scripts/frank-default-large.env`
+- Serve: `scripts/serve-frank-large.sh`
+- Chat template: `scripts/chat-template.jinja`
+
+## Known Issues
+
+1. **Patched llama.cpp required.** PR #27742 merged into the ROCmFPX fork.
+   Stock builds reject these files (TYPE_100/101 have no number in stock ggml).
+2. **Single-file payload.** Not sharded. Use `hf download` or a resumable
+   curl. Browser download unreliable for 113 GiB.
+3. **Model load is slow.** A few minutes cold for the 113 GiB file with a
+   fresh KV cache. Health returns 503 while loading. Do not kill the process.
+   `deploy.sh` waits up to 20 minutes.
+4. **Kernel spam about `amdgpu: SVM mapping failed`** is a known documented
+   symptom, not root cause. Investigate other causes before assuming memory
+   pressure.
+
+## Citations
+
+- [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
+- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
+- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture)
@@ -108,6 +108,8 @@ The LLM inference fleet on the two hosts. Full index:
 - [[freetoken-frank-default-small]] — the active frank-default-small runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver (2026-09-02)
 - [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (restored 2026-09-01)
+- [[kingjones-strix-rocmfp4]] — the active frank-default-large runbook:
+  kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B params, 113.47 GiB)
 - [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
 - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
@@ -1,5 +1,5 @@
 # Tunables for frank-default-large (kingjones777 Qwen3.8-Flash-Next
-# ROCmFP4 STRIX_LEAN, qwen4exp hybrid MoE, Dankclaw, Strix Halo gfx1151).
+# ROCmFP4 STRIX, qwen4exp hybrid MoE, Dankclaw, Strix Halo gfx1151).
 #
 # Runtime: ROCmFPX llama.cpp fork with PR #27742 (qwen4exp architecture)
 # merged and built at /home/dank/GitPackages/ROCmFPX/build-qwen38.
@@ -7,16 +7,19 @@
 # -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_CUDA_FA_ALL_QUANTS=ON
 # -DLLAMA_BUILD_WEBUI=OFF -DCMAKE_EXE_LINKER_FLAGS=-no-pie.
 # Replaced the kingjones Qwen3.8-27B Q6_0 AGENT deployment on 2026-09-01.
+# Replaced the STRIX_LEAN deployment on 2026-08-27.
 #
-# Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF.
-# 56B MoE (512 experts, 10 used), 48 layers (12 attention + 36 SSM,
-# full_attention_interval=4), 2 KV heads × 256 dim, embedding 2560,
-# QSA indexer (4 heads, key_length 128, top_k 2048), hyper-connections
+# Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF.
+# 177B params (51.2B PLE n-gram table + 125.8B total MoE/attention).
+# Qwen3.8-Flash-Next base: 56B MoE (512 experts, 10 used), 48 layers
+# (12 attention + 36 SSM, full_attention_interval=4), 2 KV heads x 256 dim,
+# embedding 2560, QSA indexer (4 heads, top_k=2048), hyper-connections
 # (count=4, low_rank=320), PLE (layer 1, ngram_size=3, heads_per_ngram=8).
-# Q4_0_ROCMFP4_STRIX_LEAN quant, 98.5 GiB, 3-part GGUF split.
+# Q4_0_ROCMFP4_STRIX quant (file_type=105), 113.47 GiB, single GGUF file.
+# PLE table: 50.66 GiB Q8_0, streamed from page cache (never in GPU VRAM).
 # Text-only (no mmproj, no MTP draft head).
 #
-# Context: 262144 (256k) × 2 slots.  f16 KV cache (~6.4 GiB per slot,
+# Context: 262144 (256k) x 2 slots.  f16 KV cache (~6.4 GiB per slot,
 # ~12.8 GiB total).  --fit off because the auto-fit code segfaults on this
 # model's hyper-connection/indexer config; -ngl 999 places all layers on
 # the SVM-backed GPU pool (116 GiB unified).
@@ -30,9 +33,9 @@
 LLAMA_BIN=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server

 # Model files from
-# https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF
-MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN
-MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-00001-of-00003.gguf
+# https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF
+MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
+MODEL_FILE=Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf

 # Qwen-Sharp chat template (tracked in the repo, staged on the host). The
 # Sharp template drives thinking on by default (medium effort); the server
@@ -46,9 +49,10 @@ ALIAS=frank-default-large
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7770

-# Serving: 262144 context (256k) × 2 slots. f16 KV, context checkpoints,
-# prompt cache. The model is 98.5 GiB loaded via mmap into the Strix Halo
-# unified memory pool (116 GiB SVM-backed). --fit off avoids the auto-fit
+# Serving: f16 KV, context checkpoints, prompt cache. The model is 113.47
+# GiB loaded via mmap into the Strix Halo unified memory pool (116 GiB
+# SVM-backed). PLE table (50.66 GiB Q8_0) is streamed from page cache; never
+# enters GPU VRAM. GPU resident: ~63-69 GiB. --fit off avoids the auto-fit
 # segfault on this model's qwen4exp hyper-connection/indexer configuration.
 CTX_SIZE=524288
 PARALLEL=2
@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN as
+# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX as
 # frank-default-large via the charlie12345/ROCmFPX llama.cpp fork with
 # PR #27742 (qwen4exp architecture) merged, on Dankclaw (Strix Halo
 # gfx1151, unified memory).
@@ -9,17 +9,17 @@
 # with -DGGML_HIP=ON -DGGML_HIP_ROCMI4_W4A4=ON
 # -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DLLAMA_BUILD_WEBUI=OFF.
 #
-# Model: 56B MoE (512 experts, 10 used), 48 layers (12 attention + 36
-# SSM/Mamba, full_attention_interval=4), 2 KV heads × 256 dim, QSA
+# Model: 177B params (56B MoE + 51.2B PLE n-gram table). 512 experts,
+# 10 used, 48 layers (12 attention + 36 SSM/Mamba,
+# full_attention_interval=4), 2 KV heads x 256 dim, QSA
 # indexer (4 heads, top_k=2048), hyper-connections (count=4, low_rank=320),
-# PLE (layer 1).  Q4_0_ROCMFP4_STRIX_LEAN quant, 98.5 GiB, 3-part split.
-#
-# Context: 131072 (128k) × 1 slot.  f16 KV cache.  Model is loaded with
-# mmap (default) — the 98.5 GiB file is memory-mapped; the Strix Halo
-# unified memory (116 GiB SVM-backed) provides the GPU address space.
+# PLE (layer 1).  Q4_0_ROCMFP4_STRIX quant, 113.47 GiB, single file.
+# PLE table (50.66 GiB Q8_0) streamed from page cache; never in GPU VRAM.
+# GPU resident: ~63-69 GiB. Context: up to 131072.
+# Loaded with mmap (default) into the Strix Halo unified memory pool.
 # --fit off is set because the auto-fit code segfaults on this model's
 # hyper-connection/indexer configuration; -ngl 999 places all layers on
-# the GPU (SVM-backed), which the unified memory pool handles.
+# the GPU (SVM-backed).
 #
 # The Qwen-Sharp chat template (scripts/chat-template.jinja) is applied at
 # runtime so the model emits reasoning in DeepSeek format; the reasoning
@@ -37,8 +37,8 @@ fi

 # --- defaults (used when the env file does not set them) -------------------
 : "${LLAMA_BIN:=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server}"
-: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN}"
-: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-00001-of-00003.gguf}"
+:: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX}"
+:: "${MODEL_FILE:=Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf}"
 : "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
 : "${CHAT_TEMPLATE_KWARGS:={}}"
 : "${ALIAS:=frank-default-large}"