serverAppConfigs/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md

---
type: reference
title: unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
description: The active frank-large runbook — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing.
tags:
  - llm
  - unsloth
  - dankclaw
  - frank-large
  - strix-halo
  - gfx1151
  - qwen4exp
  - rocm
  - mtp
timestamp: 2026-09-21
---

# unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)

> **ACTIVE since 2026-09-21.** It replaced the CIRU v4.4.1 Orca runtime
> (see [[frank-large-ciru-orca]], the current rollback target), which had
> replaced halogen-flash-server (see [[halogen-flash-server]]). Deployed as a
> **root systemd unit** (`scripts/frank-large.service`, `User=dank`), port
> 7770, on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to
> [[frank-small-gemma4-12b]].

The model is `unsloth/Qwen3.8-Flash-Next-GGUF` at **UD-IQ4_XS** (unsloth
dynamic quant): the 125B-param / 6B-active Qwen3.8-Flash-Next MoE with its
51B n-gram embedding (PLE) folded into the single GGUF, 48 layers of
12 × (3 × (Gated DeltaNet → MoE) → 1 × (QSA → MoE)), native 262,144-token
window, vision encoder. The runtime is the **unsloth prebuilt**
`app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151` — the first runtime with an
MTP graph for the qwen4exp architecture (stock ggml-org mainline has no MTP
graph, no cross-model tensor borrowing, and no `--spec-type draft-mtp` for
this model).

## Why this swap

- The operator asked for llama.cpp + the unsloth UD-IQ4_XS quant at
  **one conversation, 200k context** (2026-09-21).
- UD-IQ4_XS (87.3 GiB) is a standard llama.cpp quant — no private runtime,
  no PLE sidecar payload, no QSA hotfix: one GGUF, one MTP head, done.
- The unsloth prebuilt carries the qwen4exp MTP graph (1.3–1.7x at
  concurrency 1, per the model card), which the CIRU fork also had but which
  stock mainline does not.

## Shape

One root unit, one process, one published port:

| half | port | reachable |
|---|---|---|
| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |

## Runtime

- **Prebuilt**: `app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151.tar.gz` from
  the [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases)
  (tag `b11030-mix-5ff778e` or newer; the MTP graph landed in
  `b10715-mix-86bd2d3`). 0.4 GB, bundles its own ROCm runtime
  (`libamdhip64.so.7.16`), so there is **no host SDK to gate on and no venv
  to symlink** — unlike the CIRU predecessor's runtime-root gate.
- **Installed at** `/home/dank/GitPackages/unsloth-llama-b11030/`
  (extracted tarball; `llama-server` at the top level).
- Verified on-host (2026-09-21): `--version` → `0.4.1-dev (build 11030,
  commit 6ba30d05b)`; `--list-devices` → `ROCm0: AMD Radeon 8060S Graphics
  (116732 MiB)`; a Gemma4-12B smoke load served a clean completion on the
  iGPU before the cutover.

## Model

- **Checkpoint**: `unsloth/Qwen3.8-Flash-Next-GGUF` → `UD-IQ4_XS` (the
  `:UD-IQ4_XS` tag is a directory in the repo, not a submodel).
- **Transport**: three llama.cpp split parts (HF caps uploads at 50 GB):
  `00001` (10,946,624 B) + `00002` (49,835,229,856 B) + `00003`
  (43,836,407,744 B) = **93,682,584,224 B**. The parts ARE the canonical
  payload — the serve script points `-m` at part 1 and llama.cpp loads the
  3-way split natively (parts 2–3 auto-discovered in the same directory).
  Do **not** concatenate them into a single file to serve: part 1's header
  carries `ggml.file_count=3`, so a byte-correct concatenation still
  advertises itself as a split and the loader rejects the non-split
  filename with `invalid split file name` (caught on 2026-09-21 — the
  assembled file crash-looped the unit ~110 times before the diagnosis).
- **On disk** at `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/`:
  - `UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-0000{1,2,3}-of-00003.gguf` — the
    main model, 3-way split (87.3 GiB)
  - `mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` — MTP head (2.6 GiB)
  - `vision/mmproj-F16.gguf` — projector (904 MB, not loaded while
    `ENABLE_VISION=0`)
- **Fetch**: `scripts/download-frank-large-unsloth.sh` — resumable
  (`curl -fL -C -`), each part verified against its **LFS sha256 fetched
  live from the Hub tree API** (not a file that could drift).
  `--verify-only` re-checks without downloading.
- The n-gram embedding (PLE) is **inside the GGUF** as ordinary tensors —
  there is no external PLE payload and no pager, unlike the CIRU Orca
  packaging (79.4 GiB main + 52.4 GiB paged PLE).

## Deployed config

From `scripts/frank-large.env` (the flag list is owned by
`scripts/serve-frank-large.sh`):

| knob | value | note |
|---|---|---|
| `MODEL_DIR` | `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS` | serve script refuses to start without the GGUF and the MTP head |
| `RUNTIME_DIR` | `/home/dank/GitPackages/unsloth-llama-b11030` | prebuilt tarball, extracted |
| `HOST` / `PORT` | `0.0.0.0` / `7770` | same address the CIRU predecessor served |
| `CONTEXT_SIZE` | `200000` | operator's requested shape (2026-09-21); trained window is 262144 |
| `PARALLEL_SLOTS` | `1` | one conversation |
| `ENABLE_MTP` | `1` | the shared-Q8_0 head speculates |
| `SPEC_DRAFT_N_MAX` | `2` | the model card's default ("higher drafts more but each guess is accepted less often") |
| `ENABLE_VISION` | `0` | projector on disk, one env flip away |

Serve flags: `-m <GGUF> -md <MTP head> --spec-type draft-mtp
--spec-draft-n-max 2 -c 200000 -np 1 -ngl 999 --jinja --alias frank-large
--host 0.0.0.0 --port 7770 --chat-template-kwargs
'{"preserve_thinking":true}'`.

### The explicit `-md` (do not drop it)

Sidecar auto-discovery does **not** search the `mtp/` subfolder. Without
`-md`, `--spec-type draft-mtp` finds no head and the server runs at the main
model's speed **with no error saying why**. The serve script gates on the
head's presence and passes it explicitly.

### Memory model

The iGPU (Radeon 8060S, gfx1151) addresses **116,732 MiB (~114 GiB)** — the
GTT aperture over the shared LPDDR (`gtt_total=122402963456` in
`/sys/class/drm/card1/device/`, confirmed by `llama-server --list-devices`).
The 87.3 GiB model is mmap'd into that aperture with
`-ngl 999`, the same regime the CIRU Orca predecessor ran in (92.6 GiB
occupied at rest). Budget at 200k: weights 87.3 + MTP head 2.6 + KV ~5
(QSA 12 layers × 2 KV heads × 256 dim × f16 × 200k) + compute buffers ~5
≈ **100 GiB**, leaving ~14 GiB of aperture headroom. The host's 121 GiB of
LPDDR is shared with the CPU and the ~28 co-tenant containers; the 64 GiB
swap file is the backstop.

## Client contract
- OpenAI-compatible chat completions at `:7770/v1/chat/completions`;
  `/health` 200 when serving.
- **Reasoning** is extracted to `reasoning_content` (llama.cpp's field —
  this is a llama.cpp runtime, not vLLM's `reasoning`).
- **Tools** work via `--jinja` (the model's own template). The Qwen3.8
  template gates `reasoning_effort` to `{xhigh, medium, low}` — the harness
  folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
  `preserve_thinking` is on by default (request-level
  `chat_template_kwargs` still wins).
- **Vision**: off (`ENABLE_VISION=0`); the F16 projector is on disk. The
  fleet's image path is frank-small.
- **Audio**: none (no audio in the fleet since the FLM NPU endpoint retired
  2026-09-11).

## Troubleshooting

- **`missing runtime` / `missing target model` / `missing MTP head`** from
  the serve script: the unit exits with a pointer instead of restart-looping.
  Re-run the fetch script (`scripts/download-frank-large-unsloth.sh`) or
  re-extract the prebuilt.
- **Health 503 for minutes after (re)start**: the 87 GiB load is slow cold
  (page-cache miss). Do not kill the process; `deploy.sh` waits up to 20
  minutes (`LLM_WAIT`).
- **Serving at the main model's speed with no spec metrics**: the MTP head
  is not attached — check the process cmdline for `-md` (see above).
- **`Address already in use` on 7770**: a stray manual run of the same model
  holds the port. The deploy handler sweeps `pkill -f
  'Qwen3.8-Flash-Next-UD-IQ4_XS'` (model path, not binary name — frank-small
  is also a `llama-server`) and gates on `wait_port_free 7770`.
- **`invalid split file name` at model load** (fails in ~50 ms, before any
  weights are read): `-m` points at a file whose GGUF header advertises a
  split (`ggml.file_count > 1`) but whose name is not a
  `-NNNNN-of-NNNNN` part. This is what a concatenated split file does —
  the concatenation is byte-correct but part 1's header still says
  "3-way split". Fix: point `-m` at part 1
  (`UD-IQ4_XS/...-00001-of-00003.gguf`) and let the loader pick up parts
  2–3 from the same directory.
- **Host memory pressure**: this endpoint is the largest LPDDR consumer on
  the box. If the co-tenants start thrashing, the levers are KV quant
  (`--cache-type-k/v q8_0` halves the ~5 GiB KV) and `SPEC_DRAFT_N_MAX`
  (smaller draft state), not the weights.

## Rollback

The CIRU Orca payload and runtime are intact on disk:
`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/` (main GGUF, MTP head, PLE
payload) and `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/`.
Rollback is one commit restoring the CIRU `serve-frank-large.sh` +
`frank-large.env` + the deploy handler's `LLM_LARGE_*` paths (the
predecessor's version is in git history), then `deploy.sh dankclaw
--service frank-large`. See [[frank-large-ciru-orca]].

## References

- [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) — model card, MTP README, imatrix
- [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases) — the prebuilt runtime (tag `b11030-mix-5ff778e`)
- [MTP PR #28243](https://github.com/ggml-org/llama.cpp/pull/28243) — the qwen4exp MTP graph that the prebuilt carries