Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
323 lines
14 KiB
Markdown
323 lines
14 KiB
Markdown
---
|
||
type: reference
|
||
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw (prior, rollback target)
|
||
description: PRIOR frank-dragoon runbook and rollback target (2026-08-30 → 2026-09-11). kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
|
||
tags:
|
||
- llama.cpp
|
||
- rocm
|
||
- dankclaw
|
||
- gfx1151
|
||
- kingjones
|
||
- rocmfp4
|
||
- strix-halo
|
||
- qwen4exp
|
||
timestamp: 2026-08-30
|
||
---
|
||
|
||
# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
|
||
|
||
> **PRIOR / ROLLBACK TARGET since 2026-09-11** — frank-dragoon moved to
|
||
> [[halogen-flash-server]] (peonist-ai halogen-flash-server 0.5.8 serving
|
||
> Qwen3.8-Flash-Next w4b), which upstream measures at roughly 4x this
|
||
> runtime's prefill. Everything below documents the llama.cpp era and stays
|
||
> accurate for a rollback: the payload is still at
|
||
> `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the binary
|
||
> at `/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the unit, env and
|
||
> serve script are staged in `serverAppConfigs/.AGENTTRASH/` (`.llama.cpp`).
|
||
> Rolling back also requires removing the halogen Quadlet from
|
||
> `~/.config/containers/systemd/` so it stops claiming port 7770.
|
||
>
|
||
> Ran 2026-08-30 → 2026-09-11 on the model card's **pinned runtime**: the
|
||
> `kingjones30/ROCmFPX` fork. Text-only tier; the agentionai f16 vision
|
||
> tower was paired for mmproj and loads clean. Speculative decoding is OFF
|
||
> (the card ships no spec flags; the Flash-Next MTP heads do not load on
|
||
> this loader family — see MTP Draft Head).
|
||
|
||
`frank-dragoon` (port 7770) serves
|
||
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB)
|
||
from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/`
|
||
via `scripts/serve-frank-dragoon.sh` + `scripts/frank-dragoon.env`
|
||
and the Qwen-Sharp chat template.
|
||
|
||
## Model Architecture
|
||
|
||
Base model: Qwen3.8-Flash-Next (177B parameters total).
|
||
|
||
| Component | Params | Details |
|
||
|-----------|--------|---------|
|
||
| MoE experts | 125.8B | 512 experts, 10 used per token |
|
||
| Attention | dense | 48 layers (12 attn + 36 SSM) |
|
||
| PLE table | 51.2B | `per_layer_token_embd`, Q8_0 |
|
||
| lm head | ~0.3% | Q6_K |
|
||
|
||
The PLE (per-layer embedding) n-gram table is the only thing that differs
|
||
between STRIX, STRIX_LEAN, and FAST tiers. Everything else is byte-identical.
|
||
|
||
## File Details
|
||
|
||
| Field | Value |
|
||
|-------|-------|
|
||
| File | `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` |
|
||
| Size | 113.47 GiB |
|
||
| Format | Single GGUF (not sharded) |
|
||
| file_type | 105 (`Q4_0_ROCMFP4_STRIX`) |
|
||
| General | 113.47 GiB, 5.51 bpw |
|
||
|
||
Tensor composition:
|
||
|
||
| Tensor group | Type |
|
||
|---|---|
|
||
| MoE expert weights (`ffn_*_exps`, 144) | TYPE_101 (ROCmFP4, 4.251 bpw) |
|
||
| Shared expert (`ffn_*_shexp`, 144) | TYPE_101 |
|
||
| Attention (`attn_*`, 120) | half TYPE_100, half TYPE_101 |
|
||
| PLE (`per_layer_token_embd.weight`) | Q8_0 — 50.66 GiB |
|
||
| `token_embd.weight` | Q6_K |
|
||
| `output.weight` (lm head) | Q6_K |
|
||
| Norms / biases | F32 |
|
||
|
||
## File Type Reference
|
||
|
||
| Tier | file_type | PLE table | Total size |
|
||
|------|-----------|-----------|------------|
|
||
| FAST | TYPE_106 | 25.34 GiB | 87.94 GiB |
|
||
| STRIX_LEAN | TYPE_105 (old STRIX_LEAN) | 35.76 GiB | 98.49 GiB |
|
||
| **STRIX** | **105 (`Q4_0_ROCMFP4_STRIX`)** | **50.66 GiB** | **113.47 GiB** |
|
||
|
||
TYPE_100 = 4.506 bpw, TYPE_101 = 4.251 bpw (measured exactly by solving
|
||
three files for two unknowns).
|
||
|
||
## Memory
|
||
|
||
The PLE table never enters GPU VRAM. It streams from the file through the OS
|
||
page cache.
|
||
|
||
| Context | GTT at load | GTT after prompt | GPU resident |
|
||
|---------|-------------|-------------------|-------------|
|
||
| 8,192 | 63.3 GiB | 63.6 GiB | ~63 GiB |
|
||
| 32,768 | 64.1 GiB | 64.8 GiB | |
|
||
| 65,536 | 65.1 GiB | 66.2 GiB | |
|
||
| 131,072 | 67.2 GiB | 69.1 GiB | ~69 GiB |
|
||
|
||
~63 GiB is the non-PLE mass plus compute buffers. **The PLE table does not
|
||
grow GPU memory regardless of context depth.**
|
||
|
||
Three critical consequences:
|
||
|
||
1. **`--no-mmap` gets you OOM-killed.** It forces the PLE into anonymous memory
|
||
(neither file-backed nor reclaimable). The cgroup killer takes the process
|
||
with nothing in the server log — only `dmesg` shows the evidence.
|
||
2. **Don't force PLE to CPU with `-ot`.** Measured generation dropped from
|
||
~23 to 13.4 tok/s. The kernel streams it better than any offload hack.
|
||
3. **A bigger tier costs disk, not GPU memory.** If you have storage, take
|
||
STRIX.
|
||
|
||
## Performance
|
||
|
||
Measured on Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), 49/49 offload. Prompts
|
||
sized to 85% of context window:
|
||
|
||
| Context | Prompt tokens | pp tok/s | gen tok/s |
|
||
|---------|--------------|----------|-----------|
|
||
| 8,192 | 6,963 | 279 | **22.02** |
|
||
| 32,768 | 27,852 | 264 | **20.52** |
|
||
| 65,536 | 55,705 | 234 | **18.23** |
|
||
| 131,072 | 111,411 | 185 | **15.34** |
|
||
|
||
Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s
|
||
generation (median of 3 runs).
|
||
## Benchmark: threads-batch tuning
|
||
|
||
Sweeped `--threads-batch` (16, 32, 48, 64, 80, 96, 128) with a 64K-token
|
||
prompt via HTTP API (`max_tokens=1`). Fresh server start per value.
|
||
|
||
| threads-batch | prefill tok/s | wall time |
|
||
|---------------|--------------|-----------|
|
||
| **16** | **373.5** | 62.9s |
|
||
| 32 | 354.4 | 66.3s |
|
||
| 48 | 344.1 | 68.3s |
|
||
| 64 | 344.7 | 68.2s |
|
||
| 80 | 344.4 | 68.2s |
|
||
| 96 | 357.1 | 65.8s |
|
||
| 128 | 321.1 | 73.2s |
|
||
|
||
**Optimal: `--threads-batch 16`** (373.5 tok/s). Lower thread batching wins
|
||
for prefill on this model -- fewer threads compete for GPU resources during
|
||
the prompt processing phase. The default of 32 is 5.1% slower. Above 64,
|
||
performance plateaus then declines (128 is 14% worse than optimal).
|
||
|
||
## Benchmark: decode throughput
|
||
|
||
Sweeped `--threads-batch` (8, 16, 32) with a ~23.5K-token code prompt and
|
||
256 generated tokens via HTTP API. Fresh server start per value.
|
||
|
||
| threads-batch | gen tokens | wall time | gen tok/s |
|
||
|---------------|-----------|-----------|-----------|
|
||
| 16 | 256 | 74.8s | 3.4 |
|
||
| 32 | 256 | 75.0s | 3.4 |
|
||
| 8 | 256 | 76.6s | 3.3 |
|
||
|
||
**Decode throughput is essentially flat across all tested values.** TB=16 and
|
||
TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s.
|
||
Combined with the 5.1% prefill advantage at lower thread counts, `--threads-batch
|
||
16` is the optimal setting for both phases.
|
||
|
||
## Runtime build (kingjones30/ROCmFPX)
|
||
|
||
The card pins its own fork: [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX)
|
||
— charlie12345/ROCmFPX plus the `qwen4exp-on-rocmfpx-d3ca537` patch, so
|
||
ROCmFP4 tensor types and the qwen4exp arch live in one tree. Built at
|
||
`/home/dank/GitPackages/kingjones-ROCmFPX/build` from commit `36e9acd`
|
||
(2026-08-27 card revision) with the card's recipe:
|
||
|
||
```bash
|
||
git clone https://github.com/kingjones30/ROCmFPX.git
|
||
cd ROCmFPX
|
||
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF
|
||
cmake --build build --target llama-server llama-quantize -j16
|
||
```
|
||
|
||
- **`-DLLAMA_BUILD_WEBUI=OFF` is required on this box**: the tree
|
||
(post-WebUI llama.cpp) provisions embedded UI assets via npm or the
|
||
HF llama-ui bucket; npm is absent here and the bucket download
|
||
half-failed, aborting the link at 98%. The API server is unaffected.
|
||
All other compile work succeeded with zero errors, confirming the
|
||
card's clean-clone claim.
|
||
- All 35 flags our serve script passes were verified present in this
|
||
tree (`--spec-type`, `--cache-idle-slots`, `--ctx-checkpoints`,
|
||
`--reasoning-budget`, `--mmproj`, …).
|
||
|
||
## Serving Profile
|
||
|
||
Deployed args (`scripts/frank-dragoon.env` — our validated
|
||
profile, unchanged from the FAST deployment except the binary and model):
|
||
|
||
```
|
||
llama-server \
|
||
--model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf \
|
||
--host 0.0.0.0 --port 7770 --jinja \
|
||
--device ROCm0 --gpu-layers 999 \
|
||
--fit off --flash-attn on \
|
||
--ctx-size 401408 --parallel 2 \
|
||
--batch-size 4096 --ubatch-size 512 \
|
||
--cache-type-k f16 --cache-type-v f16 \
|
||
--threads 16 --threads-batch 16 \
|
||
--ctx-checkpoints 16 --cache-ram 8192 \
|
||
--cache-prompt --cache-idle-slots --metrics \
|
||
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
|
||
--presence-penalty 1.5 \
|
||
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
|
||
--chat-template-file chat-template.jinja \
|
||
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
|
||
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf
|
||
```
|
||
|
||
Key choices:
|
||
|
||
- **`--fit off`** — auto-fit segfaults on this model's qwen4exp
|
||
hyper-connection/indexer configuration.
|
||
- **mmap (default)** — do NOT use `--no-mmap`. See memory section.
|
||
- **`--flash-attn on` works unpatched on this model** — verified live
|
||
2026-08-30 (graph built, model served). The QSA F16 `ggml_fill`
|
||
assert that forced [[agentionai-fast-imatrix]] onto the patched
|
||
charlie tree did not fire for the kingjones STRIX file; if a future
|
||
build aborts with `GGML_ASSERT(a-type == GGML_TYPE_F32)`, port
|
||
`scripts/patches/qwen4exp-qsa-f32-mask.patch` into this tree.
|
||
- **`--ctx-size 401408 --parallel 2`** (200704/slot) — carried from the
|
||
FAST deployment; the card's own ladder proves the window is nearly
|
||
free (QSA caps KV: ~8 GiB total 8k→256k). Verified live at depth:
|
||
a 111k-token prompt prefills at ~230 t/s while a second slot decodes.
|
||
- **`--ubatch-size 512`** — card re-measurement: -ub 1024/2048/4096
|
||
show no effect; 512 is already right on this hardware.
|
||
- **`--mmproj`** — the STRIX repo ships no vision tower; the agentionai
|
||
FAST package's f16 mmproj (same base model) is paired and loads
|
||
(one benign `warmup: CLIP graph uses unsupported operators` warning).
|
||
- **No spec flags** — speculative decoding is off; see below.
|
||
|
||
## MTP Draft Head — not loadable on this loader family
|
||
|
||
The previous draft of this runbook ran
|
||
[EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF)
|
||
(Q8_0, 4.1 GB) as `--model-draft`. It does **not** load: the qwen4exp
|
||
loader requires a full per-layer `blk.<il>.hc_attn_norm.weight` set and
|
||
every Flash-Next-era head carries only `blk.48` (verified 3x against
|
||
multiple mains on 2026-08-28/29, see [[agentionai-fast-imatrix]] known
|
||
issues). The service therefore runs with no speculative decoding — the
|
||
card's own Usage block ships none. The card's published +50% decode
|
||
figure was measured with `Aristo94/EngramHalo.cpp`, a fork we do not
|
||
build. Revisit only if a head with the complete hyper-connection norm
|
||
set (or a loader that tolerates partial draft tensors) appears.
|
||
|
||
|
||
## Validation (2026-08-30 swap)
|
||
|
||
1. Integrity: on-host file `sha256sum` = `c6770d7442a06bf1d78edf28cec83e1ec93afdd34664c23ff898807b6b9349fa`,
|
||
byte-identical to the HF LFS oid. No re-download needed.
|
||
2. Load: systemd unit healthy; model loaded in ~225 s; health 200.
|
||
3. Generation: `/v1/chat/completions` returns correct
|
||
`reasoning_content` + content through the Sharp template (deepseek
|
||
format, `preserve_reasoning`).
|
||
4. Real load: an operator session's ~111k-token prompt prefilled at
|
||
213–232 t/s under the new runtime, matching the card's 131k rung
|
||
(196 t/s), with checkpoints + QSA KV streaming live.
|
||
|
||
## Deployment
|
||
|
||
### Download
|
||
|
||
```bash
|
||
# on Dankclaw
|
||
TOKEN=$(cat ~/.cache/huggingface/token)
|
||
DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
|
||
mkdir -p "$DIR"
|
||
curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
|
||
--retry 5 --retry-delay 3 --retry-max-time 7200 \
|
||
-H "Authorization: Bearer $TOKEN" \
|
||
"https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf"
|
||
```
|
||
|
||
The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
|
||
|
||
### Swap procedure (as run 2026-08-30)
|
||
|
||
1. Build the runtime (see Runtime build) BEFORE touching the service.
|
||
2. Point the model file's directory at the canonical path (a plain `mv`
|
||
— never delete a 113 GiB payload to re-fetch it).
|
||
3. Edit `scripts/frank-dragoon.env` (+ serve script defaults) via
|
||
git: commit, push, `./scripts/deploy.sh dankclaw --service
|
||
frank-dragoon`.
|
||
4. Verify health: `curl http://192.168.0.143:7770/health` and a real
|
||
`/v1/chat/completions` probe.
|
||
|
||
Rollback: the FAST deployment is one env commit away (`LLAMA_BIN`,
|
||
`MODEL_DIR`, `MODEL_FILE`, `MMPROJ_PATH`); the charlie `build-qwen38`
|
||
binary and the agentionai payload remain on disk.
|
||
|
||
### Config files
|
||
|
||
- Env: `scripts/frank-dragoon.env`
|
||
- Serve: `scripts/serve-frank-dragoon.sh`
|
||
- Chat template: `scripts/chat-template.jinja`
|
||
|
||
## Known Issues
|
||
|
||
1. **Pinned fork required.** Stock llama.cpp and plain charlie12345/
|
||
ROCmFPX each lack half the tree (qwen4exp arch / ROCmFP4 types). Use
|
||
kingjones30/ROCmFPX, or the equivalent merged charlie tree
|
||
(`/home/dank/GitPackages/ROCmFPX`, PR #27742 commits).
|
||
2. **Single-file payload.** Not sharded. Use `hf download` or a resumable
|
||
curl. Browser download unreliable for 113 GiB.
|
||
3. **Model load is slow.** A few minutes cold for the 113 GiB file with a
|
||
fresh KV cache. Health returns 503 while loading. Do not kill the process.
|
||
`deploy.sh` waits up to 20 minutes.
|
||
4. **Kernel spam about `amdgpu: SVM mapping failed`** is a known documented
|
||
symptom, not root cause. Investigate other causes before assuming memory
|
||
pressure.
|
||
|
||
## Citations
|
||
|
||
- [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
|
||
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
|
||
- [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX) — the card's pinned fork (charlie12345/ROCmFPX + [qwen4exp-on-rocmfpx-d3ca537.patch](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/qwen4exp-on-rocmfpx-d3ca537.patch), mirrored in the HF repo)
|
||
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — upstream fork defining the ROCmFP4 formats; our rollback tree with PR #27742 merged
|
||
- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support
|