docs(frank-large): unsloth Qwen3.8-Flash-Next runbook; retire the CIRU Orca runbook
New runbook frank-large-unsloth-qwen38-flash-next.md (runtime, payload, deployed config, the explicit -md, the 114 GiB GTT memory model, client contract, rollback). frank-large-ciru-orca.md marked RETIRED (rollback target, payload and runtime intact on disk). AGENTS.md topology row, frank-large section, harness note, runbook index, and the GPU-budget lesson (GTT aperture ~114 GiB, not the stale 64 GiB rocm-smi figure — the CIRU Orca held 92.6 GiB on the same device). LLM index header, mermaid, active list, and the vault root index updated. Work-log entry.
This commit is contained in:
Frank
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: CIRU v4.4.1 Orca on Dankclaw (frank-large)
|
||||
description: The active frank-large runbook — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing.
|
||||
description: The retired frank-large runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing.
|
||||
tags:
|
||||
- llm
|
||||
- ciru
|
||||
@@ -17,11 +17,13 @@ timestamp: 2026-09-19
|
||||
|
||||
# CIRU v4.4.1 Orca on Dankclaw (frank-large)
|
||||
|
||||
> **ACTIVE since 2026-09-19.** It replaced halogen-flash-server (see
|
||||
> [[halogen-flash-server]], the current rollback), which had replaced
|
||||
> ik_llama.cpp (see [[frank-large-ik-uncensored]]). Deployed as a **root
|
||||
> systemd unit** (`scripts/frank-large.service`, `User=dank`), port 7770, on the
|
||||
> same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it does not touch
|
||||
> **RETIRED 2026-09-21** (replaced by
|
||||
> [[frank-large-unsloth-qwen38-flash-next]]; **rollback target**). It was
|
||||
> ACTIVE since 2026-09-19 and replaced halogen-flash-server (see
|
||||
> [[halogen-flash-server]]), which had replaced ik_llama.cpp (see
|
||||
> [[frank-large-ik-uncensored]]). It was deployed as a **root systemd
|
||||
> unit** (`scripts/frank-large.service`, `User=dank`), port 7770, on the
|
||||
> same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it did not touch
|
||||
> belongs to [[frank-small-gemma4-12b]].
|
||||
|
||||
CIRU is a llama.cpp fork tuned for the Qwen3.8-Flash-Next ("qwen4exp")
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
---
|
||||
type: reference
|
||||
title: unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
|
||||
description: The active frank-large runbook — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing.
|
||||
tags:
|
||||
- llm
|
||||
- unsloth
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- qwen4exp
|
||||
- rocm
|
||||
- mtp
|
||||
timestamp: 2026-09-21
|
||||
---
|
||||
|
||||
# unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
|
||||
|
||||
> **ACTIVE since 2026-09-21.** It replaced the CIRU v4.4.1 Orca runtime
|
||||
> (see [[frank-large-ciru-orca]], the current rollback target), which had
|
||||
> replaced halogen-flash-server (see [[halogen-flash-server]]). Deployed as a
|
||||
> **root systemd unit** (`scripts/frank-large.service`, `User=dank`), port
|
||||
> 7770, on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to
|
||||
> [[frank-small-gemma4-12b]].
|
||||
|
||||
The model is `unsloth/Qwen3.8-Flash-Next-GGUF` at **UD-IQ4_XS** (unsloth
|
||||
dynamic quant): the 125B-param / 6B-active Qwen3.8-Flash-Next MoE with its
|
||||
51B n-gram embedding (PLE) folded into the single GGUF, 48 layers of
|
||||
12 × (3 × (Gated DeltaNet → MoE) → 1 × (QSA → MoE)), native 262,144-token
|
||||
window, vision encoder. The runtime is the **unsloth prebuilt**
|
||||
`app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151` — the first runtime with an
|
||||
MTP graph for the qwen4exp architecture (stock ggml-org mainline has no MTP
|
||||
graph, no cross-model tensor borrowing, and no `--spec-type draft-mtp` for
|
||||
this model).
|
||||
|
||||
## Why this swap
|
||||
|
||||
- The operator asked for llama.cpp + the unsloth UD-IQ4_XS quant at
|
||||
**one conversation, 200k context** (2026-09-21).
|
||||
- UD-IQ4_XS (87.3 GiB) is a standard llama.cpp quant — no private runtime,
|
||||
no PLE sidecar payload, no QSA hotfix: one GGUF, one MTP head, done.
|
||||
- The unsloth prebuilt carries the qwen4exp MTP graph (1.3–1.7x at
|
||||
concurrency 1, per the model card), which the CIRU fork also had but which
|
||||
stock mainline does not.
|
||||
|
||||
## Shape
|
||||
|
||||
One root unit, one process, one published port:
|
||||
|
||||
| half | port | reachable |
|
||||
|---|---|---|
|
||||
| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
|
||||
|
||||
## Runtime
|
||||
|
||||
- **Prebuilt**: `app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151.tar.gz` from
|
||||
the [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases)
|
||||
(tag `b11030-mix-5ff778e` or newer; the MTP graph landed in
|
||||
`b10715-mix-86bd2d3`). 0.4 GB, bundles its own ROCm runtime
|
||||
(`libamdhip64.so.7.16`), so there is **no host SDK to gate on and no venv
|
||||
to symlink** — unlike the CIRU predecessor's runtime-root gate.
|
||||
- **Installed at** `/home/dank/GitPackages/unsloth-llama-b11030/`
|
||||
(extracted tarball; `llama-server` at the top level).
|
||||
- Verified on-host (2026-09-21): `--version` → `0.4.1-dev (build 11030,
|
||||
commit 6ba30d05b)`; `--list-devices` → `ROCm0: AMD Radeon 8060S Graphics
|
||||
(116732 MiB)`; a Gemma4-12B smoke load served a clean completion on the
|
||||
iGPU before the cutover.
|
||||
|
||||
## Model
|
||||
|
||||
- **Checkpoint**: `unsloth/Qwen3.8-Flash-Next-GGUF` → `UD-IQ4_XS` (the
|
||||
`:UD-IQ4_XS` tag is a directory in the repo, not a submodel).
|
||||
- **Transport**: three llama.cpp split parts (HF caps uploads at 50 GB):
|
||||
`00001` (10,946,624 B) + `00002` (49,835,229,856 B) + `00003`
|
||||
(43,836,407,744 B) = **93,682,584,224 B** concatenated in numeric order.
|
||||
- **Assembled at** `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/`:
|
||||
- `Qwen3.8-Flash-Next-UD-IQ4_XS.gguf` — main model (87.3 GiB)
|
||||
- `mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` — MTP head (2.6 GiB)
|
||||
- `vision/mmproj-F16.gguf` — projector (904 MB, not loaded while
|
||||
`ENABLE_VISION=0`)
|
||||
- `parts/` + `SHA256SUMS` — transport parts and the assembled digest
|
||||
- **Fetch**: `scripts/download-frank-large-unsloth.sh` — resumable
|
||||
(`curl -fL -C -`), each part verified against its **LFS sha256 fetched
|
||||
live from the Hub tree API** (not a file that could drift), then the
|
||||
assembled GGUF against `SHA256SUMS`. `--verify-only` re-checks without
|
||||
downloading.
|
||||
- The n-gram embedding (PLE) is **inside the GGUF** as ordinary tensors —
|
||||
there is no external PLE payload and no pager, unlike the CIRU Orca
|
||||
packaging (79.4 GiB main + 52.4 GiB paged PLE).
|
||||
|
||||
## Deployed config
|
||||
|
||||
From `scripts/frank-large.env` (the flag list is owned by
|
||||
`scripts/serve-frank-large.sh`):
|
||||
|
||||
| knob | value | note |
|
||||
|---|---|---|
|
||||
| `MODEL_DIR` | `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS` | serve script refuses to start without the GGUF and the MTP head |
|
||||
| `RUNTIME_DIR` | `/home/dank/GitPackages/unsloth-llama-b11030` | prebuilt tarball, extracted |
|
||||
| `HOST` / `PORT` | `0.0.0.0` / `7770` | same address the CIRU predecessor served |
|
||||
| `CONTEXT_SIZE` | `200000` | operator's requested shape (2026-09-21); trained window is 262144 |
|
||||
| `PARALLEL_SLOTS` | `1` | one conversation |
|
||||
| `ENABLE_MTP` | `1` | the shared-Q8_0 head speculates |
|
||||
| `SPEC_DRAFT_N_MAX` | `2` | the model card's default ("higher drafts more but each guess is accepted less often") |
|
||||
| `ENABLE_VISION` | `0` | projector on disk, one env flip away |
|
||||
|
||||
Serve flags: `-m <GGUF> -md <MTP head> --spec-type draft-mtp
|
||||
--spec-draft-n-max 2 -c 200000 -np 1 -ngl 999 --jinja --alias frank-large
|
||||
--host 0.0.0.0 --port 7770 --chat-template-kwargs
|
||||
'{"preserve_thinking":true}'`.
|
||||
|
||||
### The explicit `-md` (do not drop it)
|
||||
|
||||
Sidecar auto-discovery does **not** search the `mtp/` subfolder. Without
|
||||
`-md`, `--spec-type draft-mtp` finds no head and the server runs at the main
|
||||
model's speed **with no error saying why**. The serve script gates on the
|
||||
head's presence and passes it explicitly.
|
||||
|
||||
### Memory model
|
||||
|
||||
The iGPU (Radeon 8060S, gfx1151) addresses **116,732 MiB (~114 GiB)** — the
|
||||
GTT aperture over the shared LPDDR (`gtt_total=122402963456` in
|
||||
`/sys/class/drm/card1/device/`, confirmed by `llama-server --list-devices`).
|
||||
The 87.3 GiB model is mmap'd into that aperture with
|
||||
`-ngl 999`, the same regime the CIRU Orca predecessor ran in (92.6 GiB
|
||||
occupied at rest). Budget at 200k: weights 87.3 + MTP head 2.6 + KV ~5
|
||||
(QSA 12 layers × 2 KV heads × 256 dim × f16 × 200k) + compute buffers ~5
|
||||
≈ **100 GiB**, leaving ~14 GiB of aperture headroom. The host's 121 GiB of
|
||||
LPDDR is shared with the CPU and the ~28 co-tenant containers; the 64 GiB
|
||||
swap file is the backstop.
|
||||
|
||||
## Client contract
|
||||
- OpenAI-compatible chat completions at `:7770/v1/chat/completions`;
|
||||
`/health` 200 when serving.
|
||||
- **Reasoning** is extracted to `reasoning_content` (llama.cpp's field —
|
||||
this is a llama.cpp runtime, not vLLM's `reasoning`).
|
||||
- **Tools** work via `--jinja` (the model's own template). The Qwen3.8
|
||||
template gates `reasoning_effort` to `{xhigh, medium, low}` — the harness
|
||||
folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
|
||||
`preserve_thinking` is on by default (request-level
|
||||
`chat_template_kwargs` still wins).
|
||||
- **Vision**: off (`ENABLE_VISION=0`); the F16 projector is on disk. The
|
||||
fleet's image path is frank-small.
|
||||
- **Audio**: none (no audio in the fleet since the FLM NPU endpoint retired
|
||||
2026-09-11).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`missing runtime` / `missing target model` / `missing MTP head`** from
|
||||
the serve script: the unit exits with a pointer instead of restart-looping.
|
||||
Re-run the fetch script (`scripts/download-frank-large-unsloth.sh`) or
|
||||
re-extract the prebuilt.
|
||||
- **Health 503 for minutes after (re)start**: the 87 GiB load is slow cold
|
||||
(page-cache miss). Do not kill the process; `deploy.sh` waits up to 20
|
||||
minutes (`LLM_WAIT`).
|
||||
- **Serving at the main model's speed with no spec metrics**: the MTP head
|
||||
is not attached — check the process cmdline for `-md` (see above).
|
||||
- **`Address already in use` on 7770**: a stray manual run of the same model
|
||||
holds the port. The deploy handler sweeps `pkill -f
|
||||
'Qwen3.8-Flash-Next-UD-IQ4_XS'` (model path, not binary name — frank-small
|
||||
is also a `llama-server`) and gates on `wait_port_free 7770`.
|
||||
- **Host memory pressure**: this endpoint is the largest LPDDR consumer on
|
||||
the box. If the co-tenants start thrashing, the levers are KV quant
|
||||
(`--cache-type-k/v q8_0` halves the ~5 GiB KV) and `SPEC_DRAFT_N_MAX`
|
||||
(smaller draft state), not the weights.
|
||||
|
||||
## Rollback
|
||||
|
||||
The CIRU Orca payload and runtime are intact on disk:
|
||||
`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/` (main GGUF, MTP head, PLE
|
||||
payload) and `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/`.
|
||||
Rollback is one commit restoring the CIRU `serve-frank-large.sh` +
|
||||
`frank-large.env` + the deploy handler's `LLM_LARGE_*` paths (the
|
||||
predecessor's version is in git history), then `deploy.sh dankclaw
|
||||
--service frank-large`. See [[frank-large-ciru-orca]].
|
||||
|
||||
## References
|
||||
|
||||
- [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) — model card, MTP README, imatrix
|
||||
- [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases) — the prebuilt runtime (tag `b11030-mix-5ff778e`)
|
||||
- [MTP PR #28243](https://github.com/ggml-org/llama.cpp/pull/28243) — the qwen4exp MTP graph that the prebuilt carries
|
||||
@@ -1,11 +1,11 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-20):
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-09-21):
|
||||
**vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
|
||||
process across both Radeon AI PRO R9700 cards (just1moremodel
|
||||
Qwen3.8-27B-Uncensored MXFP4-awq + the z-lab DFlash2-FP8 drafter, TP=2 over the
|
||||
R4D one-shot allreduce), and TWO endpoints on Dankclaw — the CIRU v4.4.1
|
||||
runtime (frank-large, jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
|
||||
process across both Radeon AI PRO R9700 cards (orcarouter
|
||||
Qwen3.8-27B-Uncensored-FP8 + the z-lab DFlash2 drafter, TP=2 over the
|
||||
R4D one-shot allreduce), and TWO endpoints on Dankclaw — the unsloth
|
||||
llama.cpp prebuilt (frank-large, Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151
|
||||
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT
|
||||
Q4_K_M with MTP and vision on the RTX 3090, port 7771).
|
||||
|
||||
@@ -23,18 +23,18 @@ graph LR
|
||||
gfx1201, 32 GiB each]
|
||||
B2[vllm-radiance 0.9.3, TP=2
|
||||
frank-medium, port 7770]
|
||||
C[Qwen3.8-27B MXFP4-awq + DFlash2-FP8
|
||||
C[Qwen3.8-27B FP8 + DFlash2 drafter
|
||||
ONE process on both cards, 262144 ctx, up to 8 seqs]
|
||||
A --> B2 --> C
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 64 GiB unified memory]
|
||||
E[CIRU v4.4.1 runtime
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[unsloth llama.cpp b11030 (ROCm gfx1151)
|
||||
frank-large, port 7770]
|
||||
F[jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b
|
||||
262144 ctx / 1 slot, MTP depth 4 + PLE]
|
||||
F[Qwen3.8-Flash-Next UD-IQ4_XS
|
||||
200000 ctx / 1 slot, MTP depth 2]
|
||||
G[mainline llama.cpp 37b53fd
|
||||
frank-small, port 7771]
|
||||
H[Gemma4-12B QAT Q4_K_M
|
||||
@@ -96,11 +96,18 @@ frank-small, port 7771]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-large-ciru-orca]] — ACTIVE frank-large runbook since 2026-09-19
|
||||
(root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build,
|
||||
QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
|
||||
iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache,
|
||||
runtime-root gate, client contract, troubleshooting
|
||||
- [[frank-large-unsloth-qwen38-flash-next]] — ACTIVE frank-large runbook
|
||||
since 2026-09-21 (root systemd unit, port 7770): the unsloth llama.cpp
|
||||
prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
|
||||
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the gfx1151 iGPU — one
|
||||
200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
|
||||
head, the 114 GiB GTT memory model, client contract, troubleshooting
|
||||
- [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by
|
||||
[[frank-large-unsloth-qwen38-flash-next]]; rollback target): the CIRU
|
||||
v4.4.1 runtime (private host build, QSA hotfix) serving
|
||||
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
|
||||
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
|
||||
gate, client contract, troubleshooting (2026-09-19 → 09-21)
|
||||
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
|
||||
[[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
|
||||
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
|
||||
@@ -161,7 +168,7 @@ frank-small, port 7771]
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on the
|
||||
CIRU v4.4.1 runtime since 2026-09-19, frank-small on the RTX 3090)
|
||||
unsloth llama.cpp prebuilt since 2026-09-21, frank-small on the RTX 3090)
|
||||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||||
|
||||
Reference in New Issue
Block a user