dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(frank-large): unsloth Qwen3.8-Flash-Next runbook; retire the CIRU Orca runbook

Browse Source 
New runbook frank-large-unsloth-qwen38-flash-next.md (runtime, payload,
deployed config, the explicit -md, the 114 GiB GTT memory model, client
contract, rollback). frank-large-ciru-orca.md marked RETIRED (rollback
target, payload and runtime intact on disk). AGENTS.md topology row,
frank-large section, harness note, runbook index, and the GPU-budget
lesson (GTT aperture ~114 GiB, not the stale 64 GiB rocm-smi figure —
the CIRU Orca held 92.6 GiB on the same device). LLM index header,
mermaid, active list, and the vault root index updated. Work-log entry.
This commit is contained in:
Frank committed 2026-09-21 12:30:35 -07:00
1 parent b149d02187
commit 7010f5b754
6 files changed
+307 -63
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md
+182
View File
@@ -0,0 +1,182 @@
---
type: reference
title: unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
description: The active frank-large runbook — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing.
tags:
- llm
- unsloth
- dankclaw
- frank-large
- strix-halo
- gfx1151
- qwen4exp
- rocm
- mtp
timestamp: 2026-09-21
---
# unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
> **ACTIVE since 2026-09-21.** It replaced the CIRU v4.4.1 Orca runtime
> (see [[frank-large-ciru-orca]], the current rollback target), which had
> replaced halogen-flash-server (see [[halogen-flash-server]]). Deployed as a
> **root systemd unit** (`scripts/frank-large.service`, `User=dank`), port
> 7770, on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to
> [[frank-small-gemma4-12b]].
The model is `unsloth/Qwen3.8-Flash-Next-GGUF` at **UD-IQ4_XS** (unsloth
dynamic quant): the 125B-param / 6B-active Qwen3.8-Flash-Next MoE with its
51B n-gram embedding (PLE) folded into the single GGUF, 48 layers of
12 × (3 × (Gated DeltaNet → MoE) → 1 × (QSA → MoE)), native 262,144-token
window, vision encoder. The runtime is the **unsloth prebuilt**
`app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151` — the first runtime with an
MTP graph for the qwen4exp architecture (stock ggml-org mainline has no MTP
graph, no cross-model tensor borrowing, and no `--spec-type draft-mtp` for
this model).
## Why this swap
- The operator asked for llama.cpp + the unsloth UD-IQ4_XS quant at
**one conversation, 200k context** (2026-09-21).
- UD-IQ4_XS (87.3 GiB) is a standard llama.cpp quant — no private runtime,
no PLE sidecar payload, no QSA hotfix: one GGUF, one MTP head, done.
- The unsloth prebuilt carries the qwen4exp MTP graph (1.3–1.7x at
concurrency 1, per the model card), which the CIRU fork also had but which
stock mainline does not.
## Shape
One root unit, one process, one published port:
| half | port | reachable |
|---|---|---|
| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
## Runtime
- **Prebuilt**: `app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151.tar.gz` from
the [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases)
(tag `b11030-mix-5ff778e` or newer; the MTP graph landed in
`b10715-mix-86bd2d3`). 0.4 GB, bundles its own ROCm runtime
(`libamdhip64.so.7.16`), so there is **no host SDK to gate on and no venv
to symlink** — unlike the CIRU predecessor's runtime-root gate.
- **Installed at** `/home/dank/GitPackages/unsloth-llama-b11030/`
(extracted tarball; `llama-server` at the top level).
- Verified on-host (2026-09-21): `--version` → `0.4.1-dev (build 11030,
commit 6ba30d05b)`; `--list-devices` → `ROCm0: AMD Radeon 8060S Graphics
(116732 MiB)`; a Gemma4-12B smoke load served a clean completion on the
iGPU before the cutover.
## Model
- **Checkpoint**: `unsloth/Qwen3.8-Flash-Next-GGUF` → `UD-IQ4_XS` (the
`:UD-IQ4_XS` tag is a directory in the repo, not a submodel).
- **Transport**: three llama.cpp split parts (HF caps uploads at 50 GB):
`00001` (10,946,624 B) + `00002` (49,835,229,856 B) + `00003`
(43,836,407,744 B) = **93,682,584,224 B** concatenated in numeric order.
- **Assembled at** `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/`:
- `Qwen3.8-Flash-Next-UD-IQ4_XS.gguf` — main model (87.3 GiB)
- `mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` — MTP head (2.6 GiB)
- `vision/mmproj-F16.gguf` — projector (904 MB, not loaded while
`ENABLE_VISION=0`)
- `parts/` + `SHA256SUMS` — transport parts and the assembled digest
- **Fetch**: `scripts/download-frank-large-unsloth.sh` — resumable
(`curl -fL -C -`), each part verified against its **LFS sha256 fetched
live from the Hub tree API** (not a file that could drift), then the
assembled GGUF against `SHA256SUMS`. `--verify-only` re-checks without
downloading.
- The n-gram embedding (PLE) is **inside the GGUF** as ordinary tensors —
there is no external PLE payload and no pager, unlike the CIRU Orca
packaging (79.4 GiB main + 52.4 GiB paged PLE).
## Deployed config
From `scripts/frank-large.env` (the flag list is owned by
`scripts/serve-frank-large.sh`):
| knob | value | note |
|---|---|---|
| `MODEL_DIR` | `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS` | serve script refuses to start without the GGUF and the MTP head |
| `RUNTIME_DIR` | `/home/dank/GitPackages/unsloth-llama-b11030` | prebuilt tarball, extracted |
| `HOST` / `PORT` | `0.0.0.0` / `7770` | same address the CIRU predecessor served |
| `CONTEXT_SIZE` | `200000` | operator's requested shape (2026-09-21); trained window is 262144 |
| `PARALLEL_SLOTS` | `1` | one conversation |
| `ENABLE_MTP` | `1` | the shared-Q8_0 head speculates |
| `SPEC_DRAFT_N_MAX` | `2` | the model card's default ("higher drafts more but each guess is accepted less often") |
| `ENABLE_VISION` | `0` | projector on disk, one env flip away |
Serve flags: `-m <GGUF> -md <MTP head> --spec-type draft-mtp
--spec-draft-n-max 2 -c 200000 -np 1 -ngl 999 --jinja --alias frank-large
--host 0.0.0.0 --port 7770 --chat-template-kwargs
'{"preserve_thinking":true}'`.
### The explicit `-md` (do not drop it)
Sidecar auto-discovery does **not** search the `mtp/` subfolder. Without
`-md`, `--spec-type draft-mtp` finds no head and the server runs at the main
model's speed **with no error saying why**. The serve script gates on the
head's presence and passes it explicitly.
### Memory model
The iGPU (Radeon 8060S, gfx1151) addresses **116,732 MiB (~114 GiB)** — the
GTT aperture over the shared LPDDR (`gtt_total=122402963456` in
`/sys/class/drm/card1/device/`, confirmed by `llama-server --list-devices`).
The 87.3 GiB model is mmap'd into that aperture with
`-ngl 999`, the same regime the CIRU Orca predecessor ran in (92.6 GiB
occupied at rest). Budget at 200k: weights 87.3 + MTP head 2.6 + KV ~5
(QSA 12 layers × 2 KV heads × 256 dim × f16 × 200k) + compute buffers ~5
≈ **100 GiB**, leaving ~14 GiB of aperture headroom. The host's 121 GiB of
LPDDR is shared with the CPU and the ~28 co-tenant containers; the 64 GiB
swap file is the backstop.
## Client contract
- OpenAI-compatible chat completions at `:7770/v1/chat/completions`;
`/health` 200 when serving.
- **Reasoning** is extracted to `reasoning_content` (llama.cpp's field —
this is a llama.cpp runtime, not vLLM's `reasoning`).
- **Tools** work via `--jinja` (the model's own template). The Qwen3.8
template gates `reasoning_effort` to `{xhigh, medium, low}` — the harness
folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
`preserve_thinking` is on by default (request-level
`chat_template_kwargs` still wins).
- **Vision**: off (`ENABLE_VISION=0`); the F16 projector is on disk. The
fleet's image path is frank-small.
- **Audio**: none (no audio in the fleet since the FLM NPU endpoint retired
2026-09-11).
## Troubleshooting
- **`missing runtime` / `missing target model` / `missing MTP head`** from
the serve script: the unit exits with a pointer instead of restart-looping.
Re-run the fetch script (`scripts/download-frank-large-unsloth.sh`) or
re-extract the prebuilt.
- **Health 503 for minutes after (re)start**: the 87 GiB load is slow cold
(page-cache miss). Do not kill the process; `deploy.sh` waits up to 20
minutes (`LLM_WAIT`).
- **Serving at the main model's speed with no spec metrics**: the MTP head
is not attached — check the process cmdline for `-md` (see above).
- **`Address already in use` on 7770**: a stray manual run of the same model
holds the port. The deploy handler sweeps `pkill -f
'Qwen3.8-Flash-Next-UD-IQ4_XS'` (model path, not binary name — frank-small
is also a `llama-server`) and gates on `wait_port_free 7770`.
- **Host memory pressure**: this endpoint is the largest LPDDR consumer on
the box. If the co-tenants start thrashing, the levers are KV quant
(`--cache-type-k/v q8_0` halves the ~5 GiB KV) and `SPEC_DRAFT_N_MAX`
(smaller draft state), not the weights.
## Rollback
The CIRU Orca payload and runtime are intact on disk:
`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/` (main GGUF, MTP head, PLE
payload) and `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/`.
Rollback is one commit restoring the CIRU `serve-frank-large.sh` +
`frank-large.env` + the deploy handler's `LLM_LARGE_*` paths (the
predecessor's version is in git history), then `deploy.sh dankclaw
--service frank-large`. See [[frank-large-ciru-orca]].
## References
- [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) — model card, MTP README, imatrix
- [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases) — the prebuilt runtime (tag `b11030-mix-5ff778e`)
- [MTP PR #28243](https://github.com/ggml-org/llama.cpp/pull/28243) — the qwen4exp MTP graph that the prebuilt carries