dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: OKF v0.1 compliance pass + fact-check + 3 research briefs

Browse Source 
- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM
- Restructure log.md to OKF (## YYYY-MM-DD, newest-first)
- Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change
- Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe)
- Write active runbook bodies from ground truth (frank-large, frank-small)
- Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026)
- Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs
- All 336 tests pass; 136 pages, 0 unresolved wikilinks
This commit is contained in:
Frank committed 2026-09-22 02:09:41 -07:00
1 parent 24f7a7772d
commit 93d1181fee
103 files changed
+2047 -4329
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md
+24 -173
View File
@@ -1,7 +1,7 @@
---
type: reference
title: unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
description: The active frank-large runbook — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing.
title: "unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)"
description: "The RETIRED frank-large runbook (2026-09-21, same-day swap; replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]) — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing."
tags:
- llm
- unsloth
@@ -15,182 +15,33 @@ tags:
timestamp: 2026-09-21
---
# unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
# frank-large — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (RETIRED)
> **ACTIVE since 2026-09-21.** It replaced the CIRU v4.4.1 Orca runtime
> (see [[frank-large-ciru-orca]], the current rollback target), which had
> replaced halogen-flash-server (see [[halogen-flash-server]]). Deployed as a
> **root systemd unit** (`scripts/frank-large.service`, `User=dank`), port
> 7770, on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to
> [[frank-small-gemma4-12b]].
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-large
> runbook on 2026-09-21 before it was replaced by
> [[frank-large-unsloth-qwen38-27b-q8xl]] (27B dense UD-Q8_K_XL) later the
> same day. The payload and runtime remain on disk as the rollback target.
The model is `unsloth/Qwen3.8-Flash-Next-GGUF` at **UD-IQ4_XS** (unsloth
dynamic quant): the 125B-param / 6B-active Qwen3.8-Flash-Next MoE with its
51B n-gram embedding (PLE) folded into the single GGUF, 48 layers of
12 × (3 × (Gated DeltaNet → MoE) → 1 × (QSA → MoE)), native 262,144-token
window, vision encoder. The runtime is the **unsloth prebuilt**
`app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151` — the first runtime with an
MTP graph for the qwen4exp architecture (stock ggml-org mainline has no MTP
graph, no cross-model tensor borrowing, and no `--spec-type draft-mtp` for
this model).
## What it served
## Why this swap
- The operator asked for llama.cpp + the unsloth UD-IQ4_XS quant at
**one conversation, 200k context** (2026-09-21).
- UD-IQ4_XS (87.3 GiB) is a standard llama.cpp quant — no private runtime,
no PLE sidecar payload, no QSA hotfix: one GGUF, one MTP head, done.
- The unsloth prebuilt carries the qwen4exp MTP graph (1.3–1.7x at
concurrency 1, per the model card), which the CIRU fork also had but which
stock mainline does not.
## Shape
One root unit, one process, one published port:
| half | port | reachable |
|---|---|---|
| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
## Runtime
- **Prebuilt**: `app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151.tar.gz` from
the [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases)
(tag `b11030-mix-5ff778e` or newer; the MTP graph landed in
`b10715-mix-86bd2d3`). 0.4 GB, bundles its own ROCm runtime
(`libamdhip64.so.7.16`), so there is **no host SDK to gate on and no venv
to symlink** — unlike the CIRU predecessor's runtime-root gate.
- **Installed at** `/home/dank/GitPackages/unsloth-llama-b11030/`
(extracted tarball; `llama-server` at the top level).
- Verified on-host (2026-09-21): `--version` → `0.4.1-dev (build 11030,
commit 6ba30d05b)`; `--list-devices` → `ROCm0: AMD Radeon 8060S Graphics
(116732 MiB)`; a Gemma4-12B smoke load served a clean completion on the
iGPU before the cutover.
## Model
- **Checkpoint**: `unsloth/Qwen3.8-Flash-Next-GGUF` → `UD-IQ4_XS` (the
`:UD-IQ4_XS` tag is a directory in the repo, not a submodel).
- **Transport**: three llama.cpp split parts (HF caps uploads at 50 GB):
`00001` (10,946,624 B) + `00002` (49,835,229,856 B) + `00003`
(43,836,407,744 B) = **93,682,584,224 B**. The parts ARE the canonical
payload — the serve script points `-m` at part 1 and llama.cpp loads the
3-way split natively (parts 2–3 auto-discovered in the same directory).
Do **not** concatenate them into a single file to serve: part 1's header
carries `ggml.file_count=3`, so a byte-correct concatenation still
advertises itself as a split and the loader rejects the non-split
filename with `invalid split file name` (caught on 2026-09-21 — the
assembled file crash-looped the unit ~110 times before the diagnosis).
- **On disk** at `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/`:
- `UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-0000{1,2,3}-of-00003.gguf` — the
main model, 3-way split (87.3 GiB)
- `mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` — MTP head (2.6 GiB)
- `vision/mmproj-F16.gguf` — projector (904 MB, not loaded while
`ENABLE_VISION=0`)
- **Fetch**: `scripts/download-frank-large-unsloth.sh` — resumable
(`curl -fL -C -`), each part verified against its **LFS sha256 fetched
live from the Hub tree API** (not a file that could drift).
`--verify-only` re-checks without downloading.
- The n-gram embedding (PLE) is **inside the GGUF** as ordinary tensors —
there is no external PLE payload and no pager, unlike the CIRU Orca
packaging (79.4 GiB main + 52.4 GiB paged PLE).
## Deployed config
From `scripts/frank-large.env` (the flag list is owned by
`scripts/serve-frank-large.sh`):
| knob | value | note |
|---|---|---|
| `MODEL_DIR` | `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS` | serve script refuses to start without the GGUF and the MTP head |
| `RUNTIME_DIR` | `/home/dank/GitPackages/unsloth-llama-b11030` | prebuilt tarball, extracted |
| `HOST` / `PORT` | `0.0.0.0` / `7770` | same address the CIRU predecessor served |
| `CONTEXT_SIZE` | `200000` | operator's requested shape (2026-09-21); trained window is 262144 |
| `PARALLEL_SLOTS` | `1` | one conversation |
| `ENABLE_MTP` | `1` | the shared-Q8_0 head speculates |
| `SPEC_DRAFT_N_MAX` | `2` | the model card's default ("higher drafts more but each guess is accepted less often") |
| `ENABLE_VISION` | `0` | projector on disk, one env flip away |
Serve flags: `-m <GGUF> -md <MTP head> --spec-type draft-mtp
--spec-draft-n-max 2 -c 200000 -np 1 -ngl 999 --jinja --alias frank-large
--host 0.0.0.0 --port 7770 --chat-template-kwargs
'{"preserve_thinking":true}'`.
### The explicit `-md` (do not drop it)
Sidecar auto-discovery does **not** search the `mtp/` subfolder. Without
`-md`, `--spec-type draft-mtp` finds no head and the server runs at the main
model's speed **with no error saying why**. The serve script gates on the
head's presence and passes it explicitly.
### Memory model
The iGPU (Radeon 8060S, gfx1151) addresses **116,732 MiB (~114 GiB)** — the
GTT aperture over the shared LPDDR (`gtt_total=122402963456` in
`/sys/class/drm/card1/device/`, confirmed by `llama-server --list-devices`).
The 87.3 GiB model is mmap'd into that aperture with
`-ngl 999`, the same regime the CIRU Orca predecessor ran in (92.6 GiB
occupied at rest). Budget at 200k: weights 87.3 + MTP head 2.6 + KV ~5
(QSA 12 layers × 2 KV heads × 256 dim × f16 × 200k) + compute buffers ~5
≈ **100 GiB**, leaving ~14 GiB of aperture headroom. The host's 121 GiB of
LPDDR is shared with the CPU and the ~28 co-tenant containers; the 64 GiB
swap file is the backstop.
## Client contract
- OpenAI-compatible chat completions at `:7770/v1/chat/completions`;
`/health` 200 when serving.
- **Reasoning** is extracted to `reasoning_content` (llama.cpp's field —
this is a llama.cpp runtime, not vLLM's `reasoning`).
- **Tools** work via `--jinja` (the model's own template). The Qwen3.8
template gates `reasoning_effort` to `{xhigh, medium, low}` — the harness
folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
`preserve_thinking` is on by default (request-level
`chat_template_kwargs` still wins).
- **Vision**: off (`ENABLE_VISION=0`); the F16 projector is on disk. The
fleet's image path is frank-small.
- **Audio**: none (no audio in the fleet since the FLM NPU endpoint retired
2026-09-11).
## Troubleshooting
- **`missing runtime` / `missing target model` / `missing MTP head`** from
the serve script: the unit exits with a pointer instead of restart-looping.
Re-run the fetch script (`scripts/download-frank-large-unsloth.sh`) or
re-extract the prebuilt.
- **Health 503 for minutes after (re)start**: the 87 GiB load is slow cold
(page-cache miss). Do not kill the process; `deploy.sh` waits up to 20
minutes (`LLM_WAIT`).
- **Serving at the main model's speed with no spec metrics**: the MTP head
is not attached — check the process cmdline for `-md` (see above).
- **`Address already in use` on 7770**: a stray manual run of the same model
holds the port. The deploy handler sweeps `pkill -f
'Qwen3.8-Flash-Next-UD-IQ4_XS'` (model path, not binary name — frank-small
is also a `llama-server`) and gates on `wait_port_free 7770`.
- **`invalid split file name` at model load** (fails in ~50 ms, before any
weights are read): `-m` points at a file whose GGUF header advertises a
split (`ggml.file_count > 1`) but whose name is not a
`-NNNNN-of-NNNNN` part. This is what a concatenated split file does —
the concatenation is byte-correct but part 1's header still says
"3-way split". Fix: point `-m` at part 1
(`UD-IQ4_XS/...-00001-of-00003.gguf`) and let the loader pick up parts
2–3 from the same directory.
- **Host memory pressure**: this endpoint is the largest LPDDR consumer on
the box. If the co-tenants start thrashing, the levers are KV quant
(`--cache-type-k/v q8_0` halves the ~5 GiB KV) and `SPEC_DRAFT_N_MAX`
(smaller draft state), not the weights.
- **Runtime:** unsloth llama.cpp prebuilt `b11030-mix-5ff778e` (ROCm
gfx1151), root systemd unit, port 7770.
- **Model:** `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (a 125B MoE / 6B
active), 87.3 GiB shipped as three split parts. The split parts ARE the
payload — concatenating them trips `invalid split file name` (part 1's
header advertises `file_count=3`).
- **Shape:** one 200,000-token conversation (single stream), MTP depth 2 via
the explicit `-md` shared-Q8_0 head. Draft acceptance verified live
0.83–0.92.
- **Memory:** the 114 GiB GTT aperture on the Strix Halo iGPU.
## Rollback
The CIRU Orca payload and runtime are intact on disk:
`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/` (main GGUF, MTP head, PLE
payload) and `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/`.
Rollback is one commit restoring the CIRU `serve-frank-large.sh` +
`frank-large.env` + the deploy handler's `LLM_LARGE_*` paths (the
predecessor's version is in git history), then `deploy.sh dankclaw
--service frank-large`. See [[frank-large-ciru-orca]].
Payload and runtime intact on disk. See [[frank-large-unsloth-qwen38-27b-q8xl]]
for the current ACTIVE runbook and [[frank-large-ciru-orca]] for the
predecessor.
## References
# Citations
- [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) — model card, MTP README, imatrix
- [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases) — the prebuilt runtime (tag `b11030-mix-5ff778e`)
- [MTP PR #28243](https://github.com/ggml-org/llama.cpp/pull/28243) — the qwen4exp MTP graph that the prebuilt carries
- unsloth prebuilts: https://huggingface.co/unsloth
- Model: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF