serverAppConfigs/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md

---

type: reference title: unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large) description: The active frank-large runbook — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing. tags:

• llm
• unsloth
• dankclaw
• frank-large
• strix-halo
• gfx1151
• qwen4exp
• rocm
• mtp timestamp: 2026-09-21

---

unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)

ACTIVE since 2026-09-21. It replaced the CIRU v4.4.1 Orca runtime (see frank-large-ciru-orca, the current rollback target), which had replaced halogen-flash-server (see halogen-flash-server). Deployed as a root systemd unit (scripts/frank-large.service, User=dank), port 7770, on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to frank-small-gemma4-12b.

The model is unsloth/Qwen3.8-Flash-Next-GGUF at UD-IQ4_XS (unsloth dynamic quant): the 125B-param / 6B-active Qwen3.8-Flash-Next MoE with its 51B n-gram embedding (PLE) folded into the single GGUF, 48 layers of 12 × (3 × (Gated DeltaNet → MoE) → 1 × (QSA → MoE)), native 262,144-token window, vision encoder. The runtime is the unsloth prebuilt app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151 — the first runtime with an MTP graph for the qwen4exp architecture (stock ggml-org mainline has no MTP graph, no cross-model tensor borrowing, and no --spec-type draft-mtp for this model).

Why this swap

• The operator asked for llama.cpp + the unsloth UD-IQ4_XS quant at one conversation, 200k context (2026-09-21).
• UD-IQ4_XS (87.3 GiB) is a standard llama.cpp quant — no private runtime, no PLE sidecar payload, no QSA hotfix: one GGUF, one MTP head, done.
• The unsloth prebuilt carries the qwen4exp MTP graph (1.3–1.7x at concurrency 1, per the model card), which the CIRU fork also had but which stock mainline does not.

Shape

One root unit, one process, one published port:

half	port	reachable
llama-server (OpenAI front-end + engine, single process)	7770	published as host 7770

Runtime

• Prebuilt: app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151.tar.gz from the unslothai/llama.cpp releases (tag b11030-mix-5ff778e or newer; the MTP graph landed in b10715-mix-86bd2d3). 0.4 GB, bundles its own ROCm runtime (libamdhip64.so.7.16), so there is no host SDK to gate on and no venv to symlink — unlike the CIRU predecessor's runtime-root gate.
• Installed at /home/dank/GitPackages/unsloth-llama-b11030/ (extracted tarball; llama-server at the top level).
• Verified on-host (2026-09-21): --version → 0.4.1-dev (build 11030, commit 6ba30d05b); --list-devices → ROCm0: AMD Radeon 8060S Graphics (116732 MiB); a Gemma4-12B smoke load served a clean completion on the iGPU before the cutover.

Model

• Checkpoint: unsloth/Qwen3.8-Flash-Next-GGUF → UD-IQ4_XS (the :UD-IQ4_XS tag is a directory in the repo, not a submodel).
• Transport: three llama.cpp split parts (HF caps uploads at 50 GB): 00001 (10,946,624 B) + 00002 (49,835,229,856 B) + 00003 (43,836,407,744 B) = 93,682,584,224 B concatenated in numeric order.
• Assembled at /home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/:
  • Qwen3.8-Flash-Next-UD-IQ4_XS.gguf — main model (87.3 GiB)
  • mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf — MTP head (2.6 GiB)
  • vision/mmproj-F16.gguf — projector (904 MB, not loaded while ENABLE_VISION=0)
  • UD-IQ4_XS/ + SHA256SUMS — transport parts and the assembled digest
• Fetch: scripts/download-frank-large-unsloth.sh — resumable (curl -fL -C -), each part verified against its LFS sha256 fetched live from the Hub tree API (not a file that could drift), then the assembled GGUF against SHA256SUMS. --verify-only re-checks without downloading.
• The n-gram embedding (PLE) is inside the GGUF as ordinary tensors — there is no external PLE payload and no pager, unlike the CIRU Orca packaging (79.4 GiB main + 52.4 GiB paged PLE).

Deployed config

From scripts/frank-large.env (the flag list is owned by scripts/serve-frank-large.sh):

knob	value	note
MODEL_DIR	/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS	serve script refuses to start without the GGUF and the MTP head
RUNTIME_DIR	/home/dank/GitPackages/unsloth-llama-b11030	prebuilt tarball, extracted
HOST / PORT	0.0.0.0 / 7770	same address the CIRU predecessor served
CONTEXT_SIZE	200000	operator's requested shape (2026-09-21); trained window is 262144
PARALLEL_SLOTS	1	one conversation
ENABLE_MTP	1	the shared-Q8_0 head speculates
SPEC_DRAFT_N_MAX	2	the model card's default ("higher drafts more but each guess is accepted less often")
ENABLE_VISION	0	projector on disk, one env flip away

Serve flags: -m <GGUF> -md <MTP head> --spec-type draft-mtp --spec-draft-n-max 2 -c 200000 -np 1 -ngl 999 --jinja --alias frank-large --host 0.0.0.0 --port 7770 --chat-template-kwargs '{"preserve_thinking":true}'.

The explicit -md (do not drop it)

Sidecar auto-discovery does not search the mtp/ subfolder. Without -md, --spec-type draft-mtp finds no head and the server runs at the main model's speed with no error saying why. The serve script gates on the head's presence and passes it explicitly.

Memory model

The iGPU (Radeon 8060S, gfx1151) addresses 116,732 MiB (~114 GiB) — the GTT aperture over the shared LPDDR (gtt_total=122402963456 in /sys/class/drm/card1/device/, confirmed by llama-server --list-devices). The 87.3 GiB model is mmap'd into that aperture with -ngl 999, the same regime the CIRU Orca predecessor ran in (92.6 GiB occupied at rest). Budget at 200k: weights 87.3 + MTP head 2.6 + KV ~5 (QSA 12 layers × 2 KV heads × 256 dim × f16 × 200k) + compute buffers ~5 ≈ 100 GiB, leaving ~14 GiB of aperture headroom. The host's 121 GiB of LPDDR is shared with the CPU and the ~28 co-tenant containers; the 64 GiB swap file is the backstop.

Client contract

• OpenAI-compatible chat completions at :7770/v1/chat/completions; /health 200 when serving.
• Reasoning is extracted to reasoning_content (llama.cpp's field — this is a llama.cpp runtime, not vLLM's reasoning).
• Tools work via --jinja (the model's own template). The Qwen3.8 template gates reasoning_effort to {xhigh, medium, low} — the harness folds its levels onto that set in models.yml (reasoningEffortMap). preserve_thinking is on by default (request-level chat_template_kwargs still wins).
• Vision: off (ENABLE_VISION=0); the F16 projector is on disk. The fleet's image path is frank-small.
• Audio: none (no audio in the fleet since the FLM NPU endpoint retired 2026-09-11).

Troubleshooting

• missing runtime / missing target model / missing MTP head from the serve script: the unit exits with a pointer instead of restart-looping. Re-run the fetch script (scripts/download-frank-large-unsloth.sh) or re-extract the prebuilt.
• Health 503 for minutes after (re)start: the 87 GiB load is slow cold (page-cache miss). Do not kill the process; deploy.sh waits up to 20 minutes (LLM_WAIT).
• Serving at the main model's speed with no spec metrics: the MTP head is not attached — check the process cmdline for -md (see above).
• Address already in use on 7770: a stray manual run of the same model holds the port. The deploy handler sweeps pkill -f 'Qwen3.8-Flash-Next-UD-IQ4_XS' (model path, not binary name — frank-small is also a llama-server) and gates on wait_port_free 7770.
• Host memory pressure: this endpoint is the largest LPDDR consumer on the box. If the co-tenants start thrashing, the levers are KV quant (--cache-type-k/v q8_0 halves the ~5 GiB KV) and SPEC_DRAFT_N_MAX (smaller draft state), not the weights.

Rollback

The CIRU Orca payload and runtime are intact on disk: /home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/ (main GGUF, MTP head, PLE payload) and /home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/. Rollback is one commit restoring the CIRU serve-frank-large.sh + frank-large.env + the deploy handler's LLM_LARGE_* paths (the predecessor's version is in git history), then deploy.sh dankclaw --service frank-large. See frank-large-ciru-orca.

References

• unsloth/Qwen3.8-Flash-Next-GGUF — model card, MTP README, imatrix
• unslothai/llama.cpp releases — the prebuilt runtime (tag b11030-mix-5ff778e)
• MTP PR #28243 — the qwen4exp MTP graph that the prebuilt carries