dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record fleet double swap (large -> kingjones STRIX on pinned fork, small -> Huihui FP8 on FreeToken)

Browse Source 
- kingjones-strix-rocmfp4 reactivated as the ACTIVE large runbook:
  runtime build recipe (WEBUI=OFF gotcha), deployed serving profile,
  fa-on-works-unpatched evidence, MTP heads documented unloadable,
  swap procedure + 2026-08-30 validation record
- agentionai-fast-imatrix marked RETIRED (rollback record kept)
- freetoken-frank-default-small: model/lineage/template sections now
  Huihui abliterated FP8; new operational quirk: chat API 422 without
  explicit model field
- LLM index + AGENTS fleet table/runtime sections current; both
  services' history chains fixed (small was still Tiel in AGENTS.md)
- worklog entry added; models.json contextWindow for large corrected
  to the real per-slot 131072 (llama clamps n_ctx_train/parallel)
This commit is contained in:
Frank committed 2026-08-30 02:35:38 -07:00
1 parent aa9c1d065a
commit 0937cfb047
6 files changed
+268 -152
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md
+99 -63
View File
@@ -1,7 +1,7 @@
---
type: reference
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
description: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the ROCmFPX llama.cpp fork. 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF. frank-default-large.
description: ACTIVE frank-default-large runbook. kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
tags:
- llama.cpp
- rocm
@@ -11,18 +11,23 @@ tags:
- rocmfp4
- strix-halo
- qwen4exp
timestamp: 2026-08-27
timestamp: 2026-08-30
---
# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
> **Active since 2026-08-27.** Replaced the STRIX_LEAN deployment on the same
> build. Same GPU memory profile, larger PLE table.
> **Active again since 2026-08-30** on the model card's **pinned runtime**:
> the `kingjones30/ROCmFPX` fork (replaced the agentionai FAST deployment
> which ran on charlie's `build-qwen38`). Text-only tier; the agentionai
> f16 vision tower is paired for mmproj and loads clean. Speculative
> decoding is OFF (the card ships no spec flags; the Flash-Next MTP heads
> do not load on this loader family — see MTP Draft Head).
`frank-default-large` (port 7770) serves
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX` from `kingjones777` on Hugging Face.
A single GGUF file, 113.47 GiB. Text-only (no mmproj). MTP draft head
loaded as a separate GGUF sidecar (see MTP Draft Head below).
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB)
from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/`
via `scripts/serve-frank-large.sh` + `scripts/frank-default-large.env`
and the Qwen-Sharp chat template.
## Model Architecture
@@ -146,12 +151,35 @@ TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s.
Combined with the 5.1% prefill advantage at lower thread counts, `--threads-batch
16` is the optimal setting for both phases.
## Runtime build (kingjones30/ROCmFPX)
The card pins its own fork: [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX)
— charlie12345/ROCmFPX plus the `qwen4exp-on-rocmfpx-d3ca537` patch, so
ROCmFP4 tensor types and the qwen4exp arch live in one tree. Built at
`/home/dank/GitPackages/kingjones-ROCmFPX/build` from commit `36e9acd`
(2026-08-27 card revision) with the card's recipe:
```bash
git clone https://github.com/kingjones30/ROCmFPX.git
cd ROCmFPX
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF
cmake --build build --target llama-server llama-quantize -j16
```
- **`-DLLAMA_BUILD_WEBUI=OFF` is required on this box**: the tree
(post-WebUI llama.cpp) provisions embedded UI assets via npm or the
HF llama-ui bucket; npm is absent here and the bucket download
half-failed, aborting the link at 98%. The API server is unaffected.
All other compile work succeeded with zero errors, confirming the
card's clean-clone claim.
- All 35 flags our serve script passes were verified present in this
tree (`--spec-type`, `--cache-idle-slots`, `--ctx-checkpoints`,
`--reasoning-budget`, `--mmproj`, …).
## Serving Profile
Runtime: charlie12345/ROCmFPX `llama-server` built at
`/home/dank/GitPackages/ROCmFPX/build-qwen38`.
Required flags:
Deployed args (`scripts/frank-default-large.env` — our validated
profile, unchanged from the FAST deployment except the binary and model):
```
llama-server \
@@ -159,8 +187,8 @@ llama-server \
--host 0.0.0.0 --port 7770 --jinja \
--device ROCm0 --gpu-layers 999 \
--fit off --flash-attn on \
--ctx-size 524288 --parallel 2 \
--batch-size 4096 --ubatch-size 4096 \
--ctx-size 401408 --parallel 2 \
--batch-size 4096 --ubatch-size 512 \
--cache-type-k f16 --cache-type-v f16 \
--threads 16 --threads-batch 16 \
--ctx-checkpoints 16 --cache-ram 8192 \
@@ -170,59 +198,57 @@ llama-server \
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
--chat-template-file chat-template.jinja \
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
--spec-type draft-mtp,ngram-mod \
--model-draft mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
--spec-draft-n-max 4 --spec-draft-p-min 0.75 \
--spec-draft-ngl 999 --spec-draft-device ROCm0 \
--spec-draft-type-k f16 --spec-draft-type-v f16
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf
```
Key choices:
- **`--fit off`** — auto-fit code segfaults on this model's qwen4exp
- **`--fit off`** — auto-fit segfaults on this model's qwen4exp
hyper-connection/indexer configuration.
- **mmap (default)** — do NOT use `--no-mmap`. See memory section.
- **`--gpu-layers 999`** — all layers on SVM-backed GPU pool.
- **`--ctx-size 524288`** — above model's max (131072); llama.cpp clamps
internally.
- **`--spec-type draft-mtp,ngram-mod`** — MTP draft head + ngram-mod
fallback. The `ngram-mod` combo is recommended by the HF card; the
prior dense 27B measurement showed ngram under MTP hurt by 4-5 t/s
([[kingjones-strix-mtp]]). Benchmark both before committing.
- **`--spec-draft-p-min 0.75`** — confidence gate. Without it, low-acceptance
prose loses more to rejected drafts than it gains (per the HF card).
- **`--flash-attn on` works unpatched on this model** — verified live
2026-08-30 (graph built, model served). The QSA F16 `ggml_fill`
assert that forced [[agentionai-fast-imatrix]] onto the patched
charlie tree did not fire for the kingjones STRIX file; if a future
build aborts with `GGML_ASSERT(a-type == GGML_TYPE_F32)`, port
`scripts/patches/qwen4exp-qsa-f32-mask.patch` into this tree.
- **`--ctx-size 401408 --parallel 2`** (200704/slot) — carried from the
FAST deployment; the card's own ladder proves the window is nearly
free (QSA caps KV: ~8 GiB total 8k→256k). Verified live at depth:
a 111k-token prompt prefills at ~230 t/s while a second slot decodes.
- **`--ubatch-size 512`** — card re-measurement: -ub 1024/2048/4096
show no effect; 512 is already right on this hardware.
- **`--mmproj`** — the STRIX repo ships no vision tower; the agentionai
FAST package's f16 mmproj (same base model) is paired and loads
(one benign `warmup: CLIP graph uses unsupported operators` warning).
- **No spec flags** — speculative decoding is off; see below.
## MTP Draft Head
## MTP Draft Head — not loadable on this loader family
The MTP (multi-token-prediction) draft head from
The previous draft of this runbook ran
[EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF)
(Q8_0, 4.1 GB). Architecture `qwen4exp`, matching the main model. Loaded
as a separate GGUF sidecar via `--model-draft`.
(Q8_0, 4.1 GB) as `--model-draft`. It does **not** load: the qwen4exp
loader requires a full per-layer `blk.<il>.hc_attn_norm.weight` set and
every Flash-Next-era head carries only `blk.48` (verified 3x against
multiple mains on 2026-08-28/29, see [[agentionai-fast-imatrix]] known
issues). The service therefore runs with no speculative decoding — the
card's own Usage block ships none. The card's published +50% decode
figure was measured with `Aristo94/EngramHalo.cpp`, a fork we do not
build. Revisit only if a head with the complete hyper-connection norm
set (or a loader that tolerates partial draft tensors) appears.
| Field | Value |
|-------|-------|
| File | `mtp-Qwen3.8-Flash-Next-Q8_0.gguf` |
| Size | 4.1 GB (4,137,429,088 B) |
| Quant | Q8_0 |
| Architecture | `qwen4exp` |
| Source | EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF |
**Measured (HF card, gfx1151, UD-IQ3_XXS target, temp 0):** decode +50% on
code (23.5 → 35.7 t/s, ~85% acceptance), +49% at 156K context depth.
Speculative decoding is lossless at temp 0. Our service runs at temp 0.7,
so expect lower acceptance; the `p_min 0.75` gate compensates.
## Validation (2026-08-30 swap)
**Caveats:**
- The MTP head is from the original `Qwen/Qwen3.8-Flash-Next`; our model is
the "Uncensored" (abliterated) variant. Abliteration modifies a small
number of weight vectors, not the full model — acceptance may be
marginally lower.
- The HF card's tested fork is `Aristo94/EngramHalo.cpp`, not our
`charlie12345/ROCmFPX`. Both have PRs #27739 (MTP) + #27742 (qwen4exp)
merged, so the MTP mechanism is the same upstream code. If the draft
load crashes, check for Strix-Halo kernel patch differences first.
- Set `SPEC_TYPE=none` in `scripts/frank-default-large.env` to disable.
1. Integrity: on-host file `sha256sum` = `c6770d7442a06bf1d78edf28cec83e1ec93afdd34664c23ff898807b6b9349fa`,
byte-identical to the HF LFS oid. No re-download needed.
2. Load: systemd unit healthy; model loaded in ~225 s; health 200.
3. Generation: `/v1/chat/completions` returns correct
`reasoning_content` + content through the Sharp template (deepseek
format, `preserve_reasoning`).
4. Real load: an operator session's ~111k-token prompt prefilled at
213–232 t/s under the new runtime, matching the card's 131k rung
(196 t/s), with checkpoints + QSA KV streaming live.
## Deployment
@@ -241,12 +267,20 @@ curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
### Swap procedure
### Swap procedure (as run 2026-08-30)
1. Stop service: `sudo systemctl stop frank-default-large`
2. Remove old model dir: `rm -rf /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN`
3. Start service: `sudo systemctl start frank-default-large`
4. Verify health: `curl http://192.168.0.143:7770/health`
1. Build the runtime (see Runtime build) BEFORE touching the service.
2. Point the model file's directory at the canonical path (a plain `mv`
— never delete a 113 GiB payload to re-fetch it).
3. Edit `scripts/frank-default-large.env` (+ serve script defaults) via
git: commit, push, `./scripts/deploy.sh dankclaw --service
frank-default-large`.
4. Verify health: `curl http://192.168.0.143:7770/health` and a real
`/v1/chat/completions` probe.
Rollback: the FAST deployment is one env commit away (`LLAMA_BIN`,
`MODEL_DIR`, `MODEL_FILE`, `MMPROJ_PATH`); the charlie `build-qwen38`
binary and the agentionai payload remain on disk.
### Config files
@@ -256,8 +290,10 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
## Known Issues
1. **Patched llama.cpp required.** PR #27742 merged into the ROCmFPX fork.
Stock builds reject these files (TYPE_100/101 have no number in stock ggml).
1. **Pinned fork required.** Stock llama.cpp and plain charlie12345/
ROCmFPX each lack half the tree (qwen4exp arch / ROCmFP4 types). Use
kingjones30/ROCmFPX, or the equivalent merged charlie tree
(`/home/dank/GitPackages/ROCmFPX`, PR #27742 commits).
2. **Single-file payload.** Not sharded. Use `hf download` or a resumable
curl. Browser download unreliable for 113 GiB.
3. **Model load is slow.** A few minutes cold for the 113 GiB file with a
@@ -271,6 +307,6 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
- [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture)
- [EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF) — MTP draft head sidecar
- [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX) — the card's pinned fork (charlie12345/ROCmFPX + [qwen4exp-on-rocmfpx-d3ca537.patch](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/qwen4exp-on-rocmfpx-d3ca537.patch), mirrored in the HF repo)
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — upstream fork defining the ROCmFP4 formats; our rollback tree with PR #27742 merged
- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support