dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f55189e880d346527bc4a0f5011397219d5859c1
serverAppConfigs/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md
T
Frank 301914b1c0 feat(llm): swap frank-large to halogen-flash-server, retire frank-npu 
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.

Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.

The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.

frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.

Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
2026-09-11 14:48:28 -07:00

323 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
type: reference
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw (prior, rollback target)
description: PRIOR frank-large runbook and rollback target (2026-08-30 → 2026-09-11). kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
tags:
- llama.cpp
- rocm
- dankclaw
- gfx1151
- kingjones
- rocmfp4
- strix-halo
- qwen4exp
timestamp: 2026-08-30
---
# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
> **PRIOR / ROLLBACK TARGET since 2026-09-11** — frank-large moved to
> [[halogen-flash-server]] (peonist-ai halogen-flash-server 0.5.8 serving
> Qwen3.8-Flash-Next w4b), which upstream measures at roughly 4x this
> runtime's prefill. Everything below documents the llama.cpp era and stays
> accurate for a rollback: the payload is still at
> `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the binary
> at `/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the unit, env and
> serve script are staged in `serverAppConfigs/.AGENTTRASH/` (`.llama.cpp`).
> Rolling back also requires removing the halogen Quadlet from
> `~/.config/containers/systemd/` so it stops claiming port 7770.
>
> Ran 2026-08-30 → 2026-09-11 on the model card's **pinned runtime**: the
> `kingjones30/ROCmFPX` fork. Text-only tier; the agentionai f16 vision
> tower was paired for mmproj and loads clean. Speculative decoding is OFF
> (the card ships no spec flags; the Flash-Next MTP heads do not load on
> this loader family — see MTP Draft Head).
`frank-large` (port 7770) serves
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB)
from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/`
via `scripts/serve-frank-large.sh` + `scripts/frank-large.env`
and the Qwen-Sharp chat template.
## Model Architecture
Base model: Qwen3.8-Flash-Next (177B parameters total).
| Component | Params | Details |
|-----------|--------|---------|
| MoE experts | 125.8B | 512 experts, 10 used per token |
| Attention | dense | 48 layers (12 attn + 36 SSM) |
| PLE table | 51.2B | `per_layer_token_embd`, Q8_0 |
| lm head | ~0.3% | Q6_K |
The PLE (per-layer embedding) n-gram table is the only thing that differs
between STRIX, STRIX_LEAN, and FAST tiers. Everything else is byte-identical.
## File Details
| Field | Value |
|-------|-------|
| File | `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` |
| Size | 113.47 GiB |
| Format | Single GGUF (not sharded) |
| file_type | 105 (`Q4_0_ROCMFP4_STRIX`) |
| General | 113.47 GiB, 5.51 bpw |
Tensor composition:
| Tensor group | Type |
|---|---|
| MoE expert weights (`ffn_*_exps`, 144) | TYPE_101 (ROCmFP4, 4.251 bpw) |
| Shared expert (`ffn_*_shexp`, 144) | TYPE_101 |
| Attention (`attn_*`, 120) | half TYPE_100, half TYPE_101 |
| PLE (`per_layer_token_embd.weight`) | Q8_0 — 50.66 GiB |
| `token_embd.weight` | Q6_K |
| `output.weight` (lm head) | Q6_K |
| Norms / biases | F32 |
## File Type Reference
| Tier | file_type | PLE table | Total size |
|------|-----------|-----------|------------|
| FAST | TYPE_106 | 25.34 GiB | 87.94 GiB |
| STRIX_LEAN | TYPE_105 (old STRIX_LEAN) | 35.76 GiB | 98.49 GiB |
| **STRIX** | **105 (`Q4_0_ROCMFP4_STRIX`)** | **50.66 GiB** | **113.47 GiB** |
TYPE_100 = 4.506 bpw, TYPE_101 = 4.251 bpw (measured exactly by solving
three files for two unknowns).
## Memory
The PLE table never enters GPU VRAM. It streams from the file through the OS
page cache.
| Context | GTT at load | GTT after prompt | GPU resident |
|---------|-------------|-------------------|-------------|
| 8,192 | 63.3 GiB | 63.6 GiB | ~63 GiB |
| 32,768 | 64.1 GiB | 64.8 GiB | |
| 65,536 | 65.1 GiB | 66.2 GiB | |
| 131,072 | 67.2 GiB | 69.1 GiB | ~69 GiB |
~63 GiB is the non-PLE mass plus compute buffers. **The PLE table does not
grow GPU memory regardless of context depth.**
Three critical consequences:
1. **`--no-mmap` gets you OOM-killed.** It forces the PLE into anonymous memory
(neither file-backed nor reclaimable). The cgroup killer takes the process
with nothing in the server log — only `dmesg` shows the evidence.
2. **Don't force PLE to CPU with `-ot`.** Measured generation dropped from
~23 to 13.4 tok/s. The kernel streams it better than any offload hack.
3. **A bigger tier costs disk, not GPU memory.** If you have storage, take
STRIX.
## Performance
Measured on Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), 49/49 offload. Prompts
sized to 85% of context window:
| Context | Prompt tokens | pp tok/s | gen tok/s |
|---------|--------------|----------|-----------|
| 8,192 | 6,963 | 279 | **22.02** |
| 32,768 | 27,852 | 264 | **20.52** |
| 65,536 | 55,705 | 234 | **18.23** |
| 131,072 | 111,411 | 185 | **15.34** |
Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s
generation (median of 3 runs).
## Benchmark: threads-batch tuning
Sweeped `--threads-batch` (16, 32, 48, 64, 80, 96, 128) with a 64K-token
prompt via HTTP API (`max_tokens=1`). Fresh server start per value.
| threads-batch | prefill tok/s | wall time |
|---------------|--------------|-----------|
| **16** | **373.5** | 62.9s |
| 32 | 354.4 | 66.3s |
| 48 | 344.1 | 68.3s |
| 64 | 344.7 | 68.2s |
| 80 | 344.4 | 68.2s |
| 96 | 357.1 | 65.8s |
| 128 | 321.1 | 73.2s |
**Optimal: `--threads-batch 16`** (373.5 tok/s). Lower thread batching wins
for prefill on this model -- fewer threads compete for GPU resources during
the prompt processing phase. The default of 32 is 5.1% slower. Above 64,
performance plateaus then declines (128 is 14% worse than optimal).
## Benchmark: decode throughput
Sweeped `--threads-batch` (8, 16, 32) with a ~23.5K-token code prompt and
256 generated tokens via HTTP API. Fresh server start per value.
| threads-batch | gen tokens | wall time | gen tok/s |
|---------------|-----------|-----------|-----------|
| 16 | 256 | 74.8s | 3.4 |
| 32 | 256 | 75.0s | 3.4 |
| 8 | 256 | 76.6s | 3.3 |
**Decode throughput is essentially flat across all tested values.** TB=16 and
TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s.
Combined with the 5.1% prefill advantage at lower thread counts, `--threads-batch
16` is the optimal setting for both phases.
## Runtime build (kingjones30/ROCmFPX)
The card pins its own fork: [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX)
— charlie12345/ROCmFPX plus the `qwen4exp-on-rocmfpx-d3ca537` patch, so
ROCmFP4 tensor types and the qwen4exp arch live in one tree. Built at
`/home/dank/GitPackages/kingjones-ROCmFPX/build` from commit `36e9acd`
(2026-08-27 card revision) with the card's recipe:
```bash
git clone https://github.com/kingjones30/ROCmFPX.git
cd ROCmFPX
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF
cmake --build build --target llama-server llama-quantize -j16
```
- **`-DLLAMA_BUILD_WEBUI=OFF` is required on this box**: the tree
(post-WebUI llama.cpp) provisions embedded UI assets via npm or the
HF llama-ui bucket; npm is absent here and the bucket download
half-failed, aborting the link at 98%. The API server is unaffected.
All other compile work succeeded with zero errors, confirming the
card's clean-clone claim.
- All 35 flags our serve script passes were verified present in this
tree (`--spec-type`, `--cache-idle-slots`, `--ctx-checkpoints`,
`--reasoning-budget`, `--mmproj`, …).
## Serving Profile
Deployed args (`scripts/frank-large.env` — our validated
profile, unchanged from the FAST deployment except the binary and model):
```
llama-server \
--model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf \
--host 0.0.0.0 --port 7770 --jinja \
--device ROCm0 --gpu-layers 999 \
--fit off --flash-attn on \
--ctx-size 401408 --parallel 2 \
--batch-size 4096 --ubatch-size 512 \
--cache-type-k f16 --cache-type-v f16 \
--threads 16 --threads-batch 16 \
--ctx-checkpoints 16 --cache-ram 8192 \
--cache-prompt --cache-idle-slots --metrics \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
--presence-penalty 1.5 \
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
--chat-template-file chat-template.jinja \
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf
```
Key choices:
- **`--fit off`** — auto-fit segfaults on this model's qwen4exp
hyper-connection/indexer configuration.
- **mmap (default)** — do NOT use `--no-mmap`. See memory section.
- **`--flash-attn on` works unpatched on this model** — verified live
2026-08-30 (graph built, model served). The QSA F16 `ggml_fill`
assert that forced [[agentionai-fast-imatrix]] onto the patched
charlie tree did not fire for the kingjones STRIX file; if a future
build aborts with `GGML_ASSERT(a-type == GGML_TYPE_F32)`, port
`scripts/patches/qwen4exp-qsa-f32-mask.patch` into this tree.
- **`--ctx-size 401408 --parallel 2`** (200704/slot) — carried from the
FAST deployment; the card's own ladder proves the window is nearly
free (QSA caps KV: ~8 GiB total 8k→256k). Verified live at depth:
a 111k-token prompt prefills at ~230 t/s while a second slot decodes.
- **`--ubatch-size 512`** — card re-measurement: -ub 1024/2048/4096
show no effect; 512 is already right on this hardware.
- **`--mmproj`** — the STRIX repo ships no vision tower; the agentionai
FAST package's f16 mmproj (same base model) is paired and loads
(one benign `warmup: CLIP graph uses unsupported operators` warning).
- **No spec flags** — speculative decoding is off; see below.
## MTP Draft Head — not loadable on this loader family
The previous draft of this runbook ran
[EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF)
(Q8_0, 4.1 GB) as `--model-draft`. It does **not** load: the qwen4exp
loader requires a full per-layer `blk.<il>.hc_attn_norm.weight` set and
every Flash-Next-era head carries only `blk.48` (verified 3x against
multiple mains on 2026-08-28/29, see [[agentionai-fast-imatrix]] known
issues). The service therefore runs with no speculative decoding — the
card's own Usage block ships none. The card's published +50% decode
figure was measured with `Aristo94/EngramHalo.cpp`, a fork we do not
build. Revisit only if a head with the complete hyper-connection norm
set (or a loader that tolerates partial draft tensors) appears.
## Validation (2026-08-30 swap)
1. Integrity: on-host file `sha256sum` = `c6770d7442a06bf1d78edf28cec83e1ec93afdd34664c23ff898807b6b9349fa`,
byte-identical to the HF LFS oid. No re-download needed.
2. Load: systemd unit healthy; model loaded in ~225 s; health 200.
3. Generation: `/v1/chat/completions` returns correct
`reasoning_content` + content through the Sharp template (deepseek
format, `preserve_reasoning`).
4. Real load: an operator session's ~111k-token prompt prefilled at
213–232 t/s under the new runtime, matching the card's 131k rung
(196 t/s), with checkpoints + QSA KV streaming live.
## Deployment
### Download
```bash
# on Dankclaw
TOKEN=$(cat ~/.cache/huggingface/token)
DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
mkdir -p "$DIR"
curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
--retry 5 --retry-delay 3 --retry-max-time 7200 \
-H "Authorization: Bearer $TOKEN" \
"https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf"
```
The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
### Swap procedure (as run 2026-08-30)
1. Build the runtime (see Runtime build) BEFORE touching the service.
2. Point the model file's directory at the canonical path (a plain `mv`
— never delete a 113 GiB payload to re-fetch it).
3. Edit `scripts/frank-large.env` (+ serve script defaults) via
git: commit, push, `./scripts/deploy.sh dankclaw --service
frank-large`.
4. Verify health: `curl http://192.168.0.143:7770/health` and a real
`/v1/chat/completions` probe.
Rollback: the FAST deployment is one env commit away (`LLAMA_BIN`,
`MODEL_DIR`, `MODEL_FILE`, `MMPROJ_PATH`); the charlie `build-qwen38`
binary and the agentionai payload remain on disk.
### Config files
- Env: `scripts/frank-large.env`
- Serve: `scripts/serve-frank-large.sh`
- Chat template: `scripts/chat-template.jinja`
## Known Issues
1. **Pinned fork required.** Stock llama.cpp and plain charlie12345/
ROCmFPX each lack half the tree (qwen4exp arch / ROCmFP4 types). Use
kingjones30/ROCmFPX, or the equivalent merged charlie tree
(`/home/dank/GitPackages/ROCmFPX`, PR #27742 commits).
2. **Single-file payload.** Not sharded. Use `hf download` or a resumable
curl. Browser download unreliable for 113 GiB.
3. **Model load is slow.** A few minutes cold for the 113 GiB file with a
fresh KV cache. Health returns 503 while loading. Do not kill the process.
`deploy.sh` waits up to 20 minutes.
4. **Kernel spam about `amdgpu: SVM mapping failed`** is a known documented
symptom, not root cause. Investigate other causes before assuming memory
pressure.
## Citations
- [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
- [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX) — the card's pinned fork (charlie12345/ROCmFPX + [qwen4exp-on-rocmfpx-d3ca537.patch](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/qwen4exp-on-rocmfpx-d3ca537.patch), mirrored in the HF repo)
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — upstream fork defining the ROCmFP4 formats; our rollback tree with PR #27742 merged
- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support
Reference in new issue View Git Blame Copy Permalink