dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
600812d459155cf1c7e0d3505f81a17e4f2a28d8
serverAppConfigs/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md
T
Frank d6f9b8539a fix(frank-large): load the model as its native 3-way split, not a concatenated file 
The first cutover crash-looped the unit with 'invalid split file name':
the fetch script concatenated the three split parts into a single GGUF,
but part 1's header carries ggml.file_count=3, so the byte-correct
concatenation still advertises itself as a split and the loader demands
a -NNNNN-of-NNNNN filename.

- serve-frank-large.sh: -m points at part 1; llama.cpp loads the split
  natively (parts 2-3 auto-discovered in the same directory)
- deploy-remote.sh: LLM_LARGE_MODEL gate path follows
- download-frank-large-unsloth.sh: assembly removed; the parts ARE the
  canonical payload, verified against live Hub LFS oids
- docs: runbook model section + troubleshooting note, AGENTS.md, log
- the redundant 87 GiB assembled file is staged on dankclaw at
  ~/.AGENTTRASH/20260921-frank-large-assembled/
2026-09-21 13:58:54 -07:00

9.9 KiB
Raw Blame History

type: reference title: unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large) description: The active frank-large runbook — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing. tags:

  • llm
  • unsloth
  • dankclaw
  • frank-large
  • strix-halo
  • gfx1151
  • qwen4exp
  • rocm
  • mtp timestamp: 2026-09-21

unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)

ACTIVE since 2026-09-21. It replaced the CIRU v4.4.1 Orca runtime (see frank-large-ciru-orca, the current rollback target), which had replaced halogen-flash-server (see halogen-flash-server). Deployed as a root systemd unit (scripts/frank-large.service, User=dank), port 7770, on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to frank-small-gemma4-12b.

The model is unsloth/Qwen3.8-Flash-Next-GGUF at UD-IQ4_XS (unsloth dynamic quant): the 125B-param / 6B-active Qwen3.8-Flash-Next MoE with its 51B n-gram embedding (PLE) folded into the single GGUF, 48 layers of 12 × (3 × (Gated DeltaNet → MoE) → 1 × (QSA → MoE)), native 262,144-token window, vision encoder. The runtime is the unsloth prebuilt app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151 — the first runtime with an MTP graph for the qwen4exp architecture (stock ggml-org mainline has no MTP graph, no cross-model tensor borrowing, and no --spec-type draft-mtp for this model).

Why this swap

  • The operator asked for llama.cpp + the unsloth UD-IQ4_XS quant at one conversation, 200k context (2026-09-21).
  • UD-IQ4_XS (87.3 GiB) is a standard llama.cpp quant — no private runtime, no PLE sidecar payload, no QSA hotfix: one GGUF, one MTP head, done.
  • The unsloth prebuilt carries the qwen4exp MTP graph (1.3–1.7x at concurrency 1, per the model card), which the CIRU fork also had but which stock mainline does not.

Shape

One root unit, one process, one published port:

half port reachable
llama-server (OpenAI front-end + engine, single process) 7770 published as host 7770

Runtime

  • Prebuilt: app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151.tar.gz from the unslothai/llama.cpp releases (tag b11030-mix-5ff778e or newer; the MTP graph landed in b10715-mix-86bd2d3). 0.4 GB, bundles its own ROCm runtime (libamdhip64.so.7.16), so there is no host SDK to gate on and no venv to symlink — unlike the CIRU predecessor's runtime-root gate.
  • Installed at /home/dank/GitPackages/unsloth-llama-b11030/ (extracted tarball; llama-server at the top level).
  • Verified on-host (2026-09-21): --version → 0.4.1-dev (build 11030, commit 6ba30d05b); --list-devices → ROCm0: AMD Radeon 8060S Graphics (116732 MiB); a Gemma4-12B smoke load served a clean completion on the iGPU before the cutover.

Model

  • Checkpoint: unsloth/Qwen3.8-Flash-Next-GGUF → UD-IQ4_XS (the :UD-IQ4_XS tag is a directory in the repo, not a submodel).
  • Transport: three llama.cpp split parts (HF caps uploads at 50 GB): 00001 (10,946,624 B) + 00002 (49,835,229,856 B) + 00003 (43,836,407,744 B) = 93,682,584,224 B. The parts ARE the canonical payload — the serve script points -m at part 1 and llama.cpp loads the 3-way split natively (parts 2–3 auto-discovered in the same directory). Do not concatenate them into a single file to serve: part 1's header carries ggml.file_count=3, so a byte-correct concatenation still advertises itself as a split and the loader rejects the non-split filename with invalid split file name (caught on 2026-09-21 — the assembled file crash-looped the unit ~110 times before the diagnosis).
  • On disk at /home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/:
    • UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-0000{1,2,3}-of-00003.gguf — the main model, 3-way split (87.3 GiB)
    • mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf — MTP head (2.6 GiB)
    • vision/mmproj-F16.gguf — projector (904 MB, not loaded while ENABLE_VISION=0)
  • Fetch: scripts/download-frank-large-unsloth.sh — resumable (curl -fL -C -), each part verified against its LFS sha256 fetched live from the Hub tree API (not a file that could drift). --verify-only re-checks without downloading.
  • The n-gram embedding (PLE) is inside the GGUF as ordinary tensors — there is no external PLE payload and no pager, unlike the CIRU Orca packaging (79.4 GiB main + 52.4 GiB paged PLE).

Deployed config

From scripts/frank-large.env (the flag list is owned by scripts/serve-frank-large.sh):

knob value note
MODEL_DIR /home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS serve script refuses to start without the GGUF and the MTP head
RUNTIME_DIR /home/dank/GitPackages/unsloth-llama-b11030 prebuilt tarball, extracted
HOST / PORT 0.0.0.0 / 7770 same address the CIRU predecessor served
CONTEXT_SIZE 200000 operator's requested shape (2026-09-21); trained window is 262144
PARALLEL_SLOTS 1 one conversation
ENABLE_MTP 1 the shared-Q8_0 head speculates
SPEC_DRAFT_N_MAX 2 the model card's default ("higher drafts more but each guess is accepted less often")
ENABLE_VISION 0 projector on disk, one env flip away

Serve flags: -m <GGUF> -md <MTP head> --spec-type draft-mtp --spec-draft-n-max 2 -c 200000 -np 1 -ngl 999 --jinja --alias frank-large --host 0.0.0.0 --port 7770 --chat-template-kwargs '{"preserve_thinking":true}'.

The explicit -md (do not drop it)

Sidecar auto-discovery does not search the mtp/ subfolder. Without -md, --spec-type draft-mtp finds no head and the server runs at the main model's speed with no error saying why. The serve script gates on the head's presence and passes it explicitly.

Memory model

The iGPU (Radeon 8060S, gfx1151) addresses 116,732 MiB (~114 GiB) — the GTT aperture over the shared LPDDR (gtt_total=122402963456 in /sys/class/drm/card1/device/, confirmed by llama-server --list-devices). The 87.3 GiB model is mmap'd into that aperture with -ngl 999, the same regime the CIRU Orca predecessor ran in (92.6 GiB occupied at rest). Budget at 200k: weights 87.3 + MTP head 2.6 + KV ~5 (QSA 12 layers × 2 KV heads × 256 dim × f16 × 200k) + compute buffers ~5 ≈ 100 GiB, leaving ~14 GiB of aperture headroom. The host's 121 GiB of LPDDR is shared with the CPU and the ~28 co-tenant containers; the 64 GiB swap file is the backstop.

Client contract

  • OpenAI-compatible chat completions at :7770/v1/chat/completions; /health 200 when serving.
  • Reasoning is extracted to reasoning_content (llama.cpp's field — this is a llama.cpp runtime, not vLLM's reasoning).
  • Tools work via --jinja (the model's own template). The Qwen3.8 template gates reasoning_effort to {xhigh, medium, low} — the harness folds its levels onto that set in models.yml (reasoningEffortMap). preserve_thinking is on by default (request-level chat_template_kwargs still wins).
  • Vision: off (ENABLE_VISION=0); the F16 projector is on disk. The fleet's image path is frank-small.
  • Audio: none (no audio in the fleet since the FLM NPU endpoint retired 2026-09-11).

Troubleshooting

  • missing runtime / missing target model / missing MTP head from the serve script: the unit exits with a pointer instead of restart-looping. Re-run the fetch script (scripts/download-frank-large-unsloth.sh) or re-extract the prebuilt.
  • Health 503 for minutes after (re)start: the 87 GiB load is slow cold (page-cache miss). Do not kill the process; deploy.sh waits up to 20 minutes (LLM_WAIT).
  • Serving at the main model's speed with no spec metrics: the MTP head is not attached — check the process cmdline for -md (see above).
  • Address already in use on 7770: a stray manual run of the same model holds the port. The deploy handler sweeps pkill -f 'Qwen3.8-Flash-Next-UD-IQ4_XS' (model path, not binary name — frank-small is also a llama-server) and gates on wait_port_free 7770.
  • invalid split file name at model load (fails in ~50 ms, before any weights are read): -m points at a file whose GGUF header advertises a split (ggml.file_count > 1) but whose name is not a -NNNNN-of-NNNNN part. This is what a concatenated split file does — the concatenation is byte-correct but part 1's header still says "3-way split". Fix: point -m at part 1 (UD-IQ4_XS/...-00001-of-00003.gguf) and let the loader pick up parts 2–3 from the same directory.
  • Host memory pressure: this endpoint is the largest LPDDR consumer on the box. If the co-tenants start thrashing, the levers are KV quant (--cache-type-k/v q8_0 halves the ~5 GiB KV) and SPEC_DRAFT_N_MAX (smaller draft state), not the weights.

Rollback

The CIRU Orca payload and runtime are intact on disk: /home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/ (main GGUF, MTP head, PLE payload) and /home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/. Rollback is one commit restoring the CIRU serve-frank-large.sh + frank-large.env + the deploy handler's LLM_LARGE_* paths (the predecessor's version is in git history), then deploy.sh dankclaw --service frank-large. See frank-large-ciru-orca.

References

Reference in new issue View Git Blame Copy Permalink