From 7010f5b7542e9ca9fff118ba7c08ae5f65da015d Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 21 Sep 2026 12:30:35 -0700 Subject: [PATCH] docs(frank-large): unsloth Qwen3.8-Flash-Next runbook; retire the CIRU Orca runbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New runbook frank-large-unsloth-qwen38-flash-next.md (runtime, payload, deployed config, the explicit -md, the 114 GiB GTT memory model, client contract, rollback). frank-large-ciru-orca.md marked RETIRED (rollback target, payload and runtime intact on disk). AGENTS.md topology row, frank-large section, harness note, runbook index, and the GPU-budget lesson (GTT aperture ~114 GiB, not the stale 64 GiB rocm-smi figure — the CIRU Orca held 92.6 GiB on the same device). LLM index header, mermaid, active list, and the vault root index updated. Work-log entry. --- AGENTS.md | 82 ++++---- .../LLM/frank-large-ciru-orca.md | 14 +- .../frank-large-unsloth-qwen38-flash-next.md | 182 ++++++++++++++++++ serverAppConfigs-docs/LLM/index.md | 39 ++-- serverAppConfigs-docs/index.md | 17 +- serverAppConfigs-docs/log.md | 36 ++++ 6 files changed, 307 insertions(+), 63 deletions(-) create mode 100644 serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md diff --git a/AGENTS.md b/AGENTS.md index 923b4f4..9ffd1d1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service. - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, - Alloy, Pi-hole) plus TWO LLM services: `frank-large` (CIRU v4.4.1 llama.cpp on + Alloy, Pi-hole) plus TWO LLM services: `frank-large` (unsloth llama.cpp prebuilt on the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack @@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with the z-lab DFlash2 external drafter (depth 7, live mean accept length 4.24/7 — A/B tested and beats MTP self-speculation on decode: 100.0 vs 85.9 t/s), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 384,238-token KV pool = 1.47x the window), fp8 KV, prefix caching, vision | 7770 | -| Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 | +| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 200000 ctx single stream (1 slot), MTP depth 2 via the explicit `-md` shared-Q8_0 head, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -265,7 +265,7 @@ a role pointing at a provider the file does not define fails at call time. As of 2026-09-20 both `frank-small` and `frank-medium` accept images (`input: [text, image]`) — frank-medium's checkpoint serves vision through the Gemma4 merge path with no separate mmproj, verified on held-out synthetic -images; `frank-large` remains text-only because its CIRU deployment loads no +images; `frank-large` remains text-only because its unsloth deployment loads no vision projector (the F16 mmproj is on disk, one env flip away). Note also that the reasoning channel differs by runtime: llama.cpp endpoints emit `reasoning_content`, vLLM emits `reasoning`. @@ -273,40 +273,45 @@ There is **no audio** input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11. -### frank-large — CIRU v4.4.1 Orca (Dankclaw, ACTIVE) +### frank-large — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (Dankclaw, ACTIVE) -CIRU v4.4.1 (the QSA window hotfix) as a **root systemd unit** (`scripts/ -frank-large.service`, `User=dank`), serving `jcbtc/Qwen3.8-Flash-CIRU-STRIX- -Orca` (79.4 GiB main GGUF + a 4.1 GiB MTP head + a 52.4 GiB PLE payload, all -sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, the model's -FULL 262,144 trained window (`PARALLEL_SLOTS=1`; the MTP drafter needs exactly -one slot in this runtime), MTP depth 4, PLE cache 4096 MiB, reasoning extracted -to `reasoning_content`, and **no vision projector** (the F16 mmproj is on disk, -`ENABLE_VISION=0` — one env flip to enable). It replaced halogen-flash-server, -which had replaced ik_llama.cpp. +The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151, +`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm runtime — +no host SDK gate, no venv) as a **root systemd unit** (`scripts/ +frank-large.service`, `User=dank`), serving +`unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB main GGUF, three HF +split parts concatenated, + a 2.6 GiB MTP head, all sha256-verified) on port +7770 from the gfx1151 iGPU. One stream, 200,000 ctx (`PARALLEL_SLOTS=1`, +operator's requested shape; the trained window is 262,144), MTP depth 2 +(`SPEC_DRAFT_N_MAX=2`, the model card's default), reasoning extracted to +`reasoning_content`, and **no vision projector** (the F16 mmproj is on disk, +`ENABLE_VISION=0` — one env flip to enable). It replaced the CIRU v4.4.1 +Orca runtime, which had replaced halogen-flash-server. -The runtime is a **private host build** (`/home/dank/GitPackages/ciru-runtime- -v4.4.1-qsa-hotfix/`), not a container image: a gfx1151 HIP build of the CIRU -llama.cpp fork with its own bundled ROCm SDK in a venv (`.venv-rocm/`), and the -binary resolves the SDK libs via RUNPATH. The launcher (`scripts/ciru/ -run-orca-server.sh` → `run-server.sh`) has a **runtime gate**: it exits 2 unless -`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/ -libhsa-runtime64.so` exist. The serve script (`scripts/serve-frank-large.sh`) -satisfies that gate idempotently by `ln -sfn`-ing both to the venv's unversioned -devel symlinks, then `exec`s the launcher with `--chat-template-kwargs -'{"preserve_thinking":true}'` (Qwen3.8's template needs this to carry thinking -blocks across turns; a request-level value still wins). +The runtime is the first with an MTP graph for the qwen4exp architecture +(stock ggml-org mainline has none), so the unsloth prebuilt is the +llama.cpp-compatible choice. The MTP head is loaded via an **explicit +`-md`**: sidecar auto-discovery does not search the `mtp/` subfolder, and +without it `--spec-type draft-mtp` silently runs at the main model's speed. +The serve script (`scripts/serve-frank-large.sh`) validates binary → main +GGUF → MTP head and exits with a pointer message if any is missing, then +`exec`s `llama-server` with `--jinja --alias frank-large --chat-template- +kwargs '{"preserve_thinking":true}'` (Qwen3.8's template needs this to +carry thinking blocks across turns; a request-level value still wins). -The serve script validates launcher → built binary → venv runtime → main GGUF → -MTP head → PLE payload + manifest + scale, and exits with a pointer message if -any is missing. `PLE_CACHE_MIB=4096` is the vendor production-profile default; -the 52.4 GiB PLE is mmap'd and paged, not resident. +The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR, +`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated" +is not the budget). The 87.3 GiB model is mmap'd in with `-ngl 999`; budget +at 200k ≈ 100 GiB (weights + MTP head + KV + buffers), leaving ~14 GiB of +aperture headroom. The n-gram embedding (PLE) is inside the GGUF — no +external PLE payload or pager, unlike the CIRU Orca packaging. -The halogen Quadlet (`dankclaw/frank-large/quadlet/`) was retired to -`.AGENTTRASH/` — one owner per endpoint, now the scripts/ trio. +Fetch: `scripts/download-frank-large-unsloth.sh` (resumable, each part +verified against its LFS sha256 fetched live from the Hub tree API, then the +assembled GGUF against `SHA256SUMS`; `--verify-only` re-checks). -Runbook: `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md`. -Rollback: [[halogen-flash-server]]. +Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md`. +Rollback: [[frank-large-ciru-orca]] (payload and runtime intact on disk). ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) @@ -530,9 +535,13 @@ linked runbooks. - **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on the agentionai FAST layer set; the kingjones STRIX set works unpatched. -- **GPU budget on Dankclaw is the rocm-smi figure** (64 GiB, SVM-backed). - `free`/`dmidecode` show the CPU-side carve-out — intentional, do not - change. +- **GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the + rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports + 116,732 MiB (`llama-server --list-devices`; + `gtt_total=122402963456` in `/sys/class/drm/card1/device/`). The older + "64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor + held 92.6 GiB of weights on the iGPU. `free`/`dmidecode` show the + CPU-side carve-out — intentional, do not change. **Load / crash diagnosis** @@ -743,7 +752,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` | +| frank-large (unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | | frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` | | frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` | @@ -757,6 +766,7 @@ Retired (rollback targets and history): | kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` | | halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` | +| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` | | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` | | vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | diff --git a/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md b/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md index 623d85d..e6a7826 100644 --- a/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md +++ b/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md @@ -1,7 +1,7 @@ --- type: reference title: CIRU v4.4.1 Orca on Dankclaw (frank-large) -description: The active frank-large runbook — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing. +description: The retired frank-large runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing. tags: - llm - ciru @@ -17,11 +17,13 @@ timestamp: 2026-09-19 # CIRU v4.4.1 Orca on Dankclaw (frank-large) -> **ACTIVE since 2026-09-19.** It replaced halogen-flash-server (see -> [[halogen-flash-server]], the current rollback), which had replaced -> ik_llama.cpp (see [[frank-large-ik-uncensored]]). Deployed as a **root -> systemd unit** (`scripts/frank-large.service`, `User=dank`), port 7770, on the -> same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it does not touch +> **RETIRED 2026-09-21** (replaced by +> [[frank-large-unsloth-qwen38-flash-next]]; **rollback target**). It was +> ACTIVE since 2026-09-19 and replaced halogen-flash-server (see +> [[halogen-flash-server]]), which had replaced ik_llama.cpp (see +> [[frank-large-ik-uncensored]]). It was deployed as a **root systemd +> unit** (`scripts/frank-large.service`, `User=dank`), port 7770, on the +> same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it did not touch > belongs to [[frank-small-gemma4-12b]]. CIRU is a llama.cpp fork tuned for the Qwen3.8-Flash-Next ("qwen4exp") diff --git a/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md b/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md new file mode 100644 index 0000000..8f8580a --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md @@ -0,0 +1,182 @@ +--- +type: reference +title: unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large) +description: The active frank-large runbook — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing. +tags: + - llm + - unsloth + - dankclaw + - frank-large + - strix-halo + - gfx1151 + - qwen4exp + - rocm + - mtp +timestamp: 2026-09-21 +--- + +# unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large) + +> **ACTIVE since 2026-09-21.** It replaced the CIRU v4.4.1 Orca runtime +> (see [[frank-large-ciru-orca]], the current rollback target), which had +> replaced halogen-flash-server (see [[halogen-flash-server]]). Deployed as a +> **root systemd unit** (`scripts/frank-large.service`, `User=dank`), port +> 7770, on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to +> [[frank-small-gemma4-12b]]. + +The model is `unsloth/Qwen3.8-Flash-Next-GGUF` at **UD-IQ4_XS** (unsloth +dynamic quant): the 125B-param / 6B-active Qwen3.8-Flash-Next MoE with its +51B n-gram embedding (PLE) folded into the single GGUF, 48 layers of +12 × (3 × (Gated DeltaNet → MoE) → 1 × (QSA → MoE)), native 262,144-token +window, vision encoder. The runtime is the **unsloth prebuilt** +`app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151` — the first runtime with an +MTP graph for the qwen4exp architecture (stock ggml-org mainline has no MTP +graph, no cross-model tensor borrowing, and no `--spec-type draft-mtp` for +this model). + +## Why this swap + +- The operator asked for llama.cpp + the unsloth UD-IQ4_XS quant at + **one conversation, 200k context** (2026-09-21). +- UD-IQ4_XS (87.3 GiB) is a standard llama.cpp quant — no private runtime, + no PLE sidecar payload, no QSA hotfix: one GGUF, one MTP head, done. +- The unsloth prebuilt carries the qwen4exp MTP graph (1.3–1.7x at + concurrency 1, per the model card), which the CIRU fork also had but which + stock mainline does not. + +## Shape + +One root unit, one process, one published port: + +| half | port | reachable | +|---|---|---| +| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** | + +## Runtime + +- **Prebuilt**: `app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151.tar.gz` from + the [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases) + (tag `b11030-mix-5ff778e` or newer; the MTP graph landed in + `b10715-mix-86bd2d3`). 0.4 GB, bundles its own ROCm runtime + (`libamdhip64.so.7.16`), so there is **no host SDK to gate on and no venv + to symlink** — unlike the CIRU predecessor's runtime-root gate. +- **Installed at** `/home/dank/GitPackages/unsloth-llama-b11030/` + (extracted tarball; `llama-server` at the top level). +- Verified on-host (2026-09-21): `--version` → `0.4.1-dev (build 11030, + commit 6ba30d05b)`; `--list-devices` → `ROCm0: AMD Radeon 8060S Graphics + (116732 MiB)`; a Gemma4-12B smoke load served a clean completion on the + iGPU before the cutover. + +## Model + +- **Checkpoint**: `unsloth/Qwen3.8-Flash-Next-GGUF` → `UD-IQ4_XS` (the + `:UD-IQ4_XS` tag is a directory in the repo, not a submodel). +- **Transport**: three llama.cpp split parts (HF caps uploads at 50 GB): + `00001` (10,946,624 B) + `00002` (49,835,229,856 B) + `00003` + (43,836,407,744 B) = **93,682,584,224 B** concatenated in numeric order. +- **Assembled at** `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/`: + - `Qwen3.8-Flash-Next-UD-IQ4_XS.gguf` — main model (87.3 GiB) + - `mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` — MTP head (2.6 GiB) + - `vision/mmproj-F16.gguf` — projector (904 MB, not loaded while + `ENABLE_VISION=0`) + - `parts/` + `SHA256SUMS` — transport parts and the assembled digest +- **Fetch**: `scripts/download-frank-large-unsloth.sh` — resumable + (`curl -fL -C -`), each part verified against its **LFS sha256 fetched + live from the Hub tree API** (not a file that could drift), then the + assembled GGUF against `SHA256SUMS`. `--verify-only` re-checks without + downloading. +- The n-gram embedding (PLE) is **inside the GGUF** as ordinary tensors — + there is no external PLE payload and no pager, unlike the CIRU Orca + packaging (79.4 GiB main + 52.4 GiB paged PLE). + +## Deployed config + +From `scripts/frank-large.env` (the flag list is owned by +`scripts/serve-frank-large.sh`): + +| knob | value | note | +|---|---|---| +| `MODEL_DIR` | `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS` | serve script refuses to start without the GGUF and the MTP head | +| `RUNTIME_DIR` | `/home/dank/GitPackages/unsloth-llama-b11030` | prebuilt tarball, extracted | +| `HOST` / `PORT` | `0.0.0.0` / `7770` | same address the CIRU predecessor served | +| `CONTEXT_SIZE` | `200000` | operator's requested shape (2026-09-21); trained window is 262144 | +| `PARALLEL_SLOTS` | `1` | one conversation | +| `ENABLE_MTP` | `1` | the shared-Q8_0 head speculates | +| `SPEC_DRAFT_N_MAX` | `2` | the model card's default ("higher drafts more but each guess is accepted less often") | +| `ENABLE_VISION` | `0` | projector on disk, one env flip away | + +Serve flags: `-m -md --spec-type draft-mtp +--spec-draft-n-max 2 -c 200000 -np 1 -ngl 999 --jinja --alias frank-large +--host 0.0.0.0 --port 7770 --chat-template-kwargs +'{"preserve_thinking":true}'`. + +### The explicit `-md` (do not drop it) + +Sidecar auto-discovery does **not** search the `mtp/` subfolder. Without +`-md`, `--spec-type draft-mtp` finds no head and the server runs at the main +model's speed **with no error saying why**. The serve script gates on the +head's presence and passes it explicitly. + +### Memory model + +The iGPU (Radeon 8060S, gfx1151) addresses **116,732 MiB (~114 GiB)** — the +GTT aperture over the shared LPDDR (`gtt_total=122402963456` in +`/sys/class/drm/card1/device/`, confirmed by `llama-server --list-devices`). +The 87.3 GiB model is mmap'd into that aperture with +`-ngl 999`, the same regime the CIRU Orca predecessor ran in (92.6 GiB +occupied at rest). Budget at 200k: weights 87.3 + MTP head 2.6 + KV ~5 +(QSA 12 layers × 2 KV heads × 256 dim × f16 × 200k) + compute buffers ~5 +≈ **100 GiB**, leaving ~14 GiB of aperture headroom. The host's 121 GiB of +LPDDR is shared with the CPU and the ~28 co-tenant containers; the 64 GiB +swap file is the backstop. + +## Client contract +- OpenAI-compatible chat completions at `:7770/v1/chat/completions`; + `/health` 200 when serving. +- **Reasoning** is extracted to `reasoning_content` (llama.cpp's field — + this is a llama.cpp runtime, not vLLM's `reasoning`). +- **Tools** work via `--jinja` (the model's own template). The Qwen3.8 + template gates `reasoning_effort` to `{xhigh, medium, low}` — the harness + folds its levels onto that set in `models.yml` (`reasoningEffortMap`). + `preserve_thinking` is on by default (request-level + `chat_template_kwargs` still wins). +- **Vision**: off (`ENABLE_VISION=0`); the F16 projector is on disk. The + fleet's image path is frank-small. +- **Audio**: none (no audio in the fleet since the FLM NPU endpoint retired + 2026-09-11). + +## Troubleshooting + +- **`missing runtime` / `missing target model` / `missing MTP head`** from + the serve script: the unit exits with a pointer instead of restart-looping. + Re-run the fetch script (`scripts/download-frank-large-unsloth.sh`) or + re-extract the prebuilt. +- **Health 503 for minutes after (re)start**: the 87 GiB load is slow cold + (page-cache miss). Do not kill the process; `deploy.sh` waits up to 20 + minutes (`LLM_WAIT`). +- **Serving at the main model's speed with no spec metrics**: the MTP head + is not attached — check the process cmdline for `-md` (see above). +- **`Address already in use` on 7770**: a stray manual run of the same model + holds the port. The deploy handler sweeps `pkill -f + 'Qwen3.8-Flash-Next-UD-IQ4_XS'` (model path, not binary name — frank-small + is also a `llama-server`) and gates on `wait_port_free 7770`. +- **Host memory pressure**: this endpoint is the largest LPDDR consumer on + the box. If the co-tenants start thrashing, the levers are KV quant + (`--cache-type-k/v q8_0` halves the ~5 GiB KV) and `SPEC_DRAFT_N_MAX` + (smaller draft state), not the weights. + +## Rollback + +The CIRU Orca payload and runtime are intact on disk: +`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/` (main GGUF, MTP head, PLE +payload) and `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/`. +Rollback is one commit restoring the CIRU `serve-frank-large.sh` + +`frank-large.env` + the deploy handler's `LLM_LARGE_*` paths (the +predecessor's version is in git history), then `deploy.sh dankclaw +--service frank-large`. See [[frank-large-ciru-orca]]. + +## References + +- [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) — model card, MTP README, imatrix +- [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases) — the prebuilt runtime (tag `b11030-mix-5ff778e`) +- [MTP PR #28243](https://github.com/ggml-org/llama.cpp/pull/28243) — the qwen4exp MTP graph that the prebuilt carries diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index a837fb6..14af88f 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,11 +1,11 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on the two hosts (2026-09-20): +Documentation for the LLM inference fleet on the two hosts (2026-09-21): **vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel -process across both Radeon AI PRO R9700 cards (just1moremodel -Qwen3.8-27B-Uncensored MXFP4-awq + the z-lab DFlash2-FP8 drafter, TP=2 over the -R4D one-shot allreduce), and TWO endpoints on Dankclaw — the CIRU v4.4.1 -runtime (frank-large, jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 +process across both Radeon AI PRO R9700 cards (orcarouter +Qwen3.8-27B-Uncensored-FP8 + the z-lab DFlash2 drafter, TP=2 over the +R4D one-shot allreduce), and TWO endpoints on Dankclaw — the unsloth +llama.cpp prebuilt (frank-large, Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771). @@ -23,18 +23,18 @@ graph LR gfx1201, 32 GiB each] B2[vllm-radiance 0.9.3, TP=2 frank-medium, port 7770] - C[Qwen3.8-27B MXFP4-awq + DFlash2-FP8 + C[Qwen3.8-27B FP8 + DFlash2 drafter ONE process on both cards, 262144 ctx, up to 8 seqs] A --> B2 --> C end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] E2[RTX 3090 (PCIe x4 dock) + -gfx1151 iGPU 64 GiB unified memory] - E[CIRU v4.4.1 runtime +gfx1151 iGPU 114 GiB GTT] + E[unsloth llama.cpp b11030 (ROCm gfx1151) frank-large, port 7770] - F[jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b -262144 ctx / 1 slot, MTP depth 4 + PLE] + F[Qwen3.8-Flash-Next UD-IQ4_XS +200000 ctx / 1 slot, MTP depth 2] G[mainline llama.cpp 37b53fd frank-small, port 7771] H[Gemma4-12B QAT Q4_K_M @@ -96,11 +96,18 @@ frank-small, port 7771] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-large-ciru-orca]] — ACTIVE frank-large runbook since 2026-09-19 - (root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build, - QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 - iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, - runtime-root gate, client contract, troubleshooting +- [[frank-large-unsloth-qwen38-flash-next]] — ACTIVE frank-large runbook + since 2026-09-21 (root systemd unit, port 7770): the unsloth llama.cpp + prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving + unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the gfx1151 iGPU — one + 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0 + head, the 114 GiB GTT memory model, client contract, troubleshooting +- [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by + [[frank-large-unsloth-qwen38-flash-next]]; rollback target): the CIRU + v4.4.1 runtime (private host build, QSA hotfix) serving + jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one + 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root + gate, client contract, troubleshooting (2026-09-19 → 09-21) - [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by [[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b @@ -161,7 +168,7 @@ frank-small, port 7771] (deployed and reverted 2026-08-22) - [[Dankserver System Specs]] — the 2x R9700 box (frank-medium) - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on the - CIRU v4.4.1 runtime since 2026-09-19, frank-small on the RTX 3090) + unsloth llama.cpp prebuilt since 2026-09-21, frank-small on the RTX 3090) - [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as the retirement record — the IOMMU/SVA err=-19 root cause and where diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 9da7dd7..63717e4 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -147,11 +147,18 @@ The LLM inference fleet on the two hosts. Full index: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record) -- [[frank-large-ciru-orca]] — the ACTIVE frank-large runbook since 2026-09-19 - (root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build, - QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 - iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, - runtime-root gate, client contract +- [[frank-large-unsloth-qwen38-flash-next]] — the ACTIVE frank-large runbook + since 2026-09-21 (root systemd unit, port 7770): the unsloth llama.cpp + prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving + unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the gfx1151 iGPU — one + 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0 + head, the 114 GiB GTT memory model, client contract +- [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by + [[frank-large-unsloth-qwen38-flash-next]]; rollback target, 2026-09-19 → + 09-21): the CIRU v4.4.1 runtime (private host build, QSA hotfix) serving + jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one + 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root + gate, client contract - [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by [[frank-large-ciru-orca]]; rollback target, 2026-09-16 → 09-19): peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8- diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 2bfa9c5..122ed83 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -11,6 +11,42 @@ timestamp: 2026-09-14 # Bundle Change Log +## 2026-09-21 — frank-large: CIRU Orca → unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS + +- **The change.** At the operator's request, frank-large (dankclaw :7770) + moves from the CIRU v4.4.1 Orca runtime (jcbtc + Qwen3.8-Flash-CIRU-STRIX-Orca, 262144 ctx) to the unsloth llama.cpp + prebuilt (b11030-mix-5ff778e, ROCm gfx1151, bundles its own ROCm 7.16) + serving `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB, three HF + split parts concatenated), one conversation at 200,000 ctx, MTP depth 2. + The fleet contract is unchanged: port 7770, alias `frank-large`, + text-only. +- **Why the unsloth prebuilt.** The qwen4exp MTP graph exists only in the + unsloth fork (stock mainline has no MTP for this architecture); the + prebuilt bundles its own ROCm, so there is no host SDK gate or venv the + way the CIRU build had one. +- **The explicit `-md`.** Sidecar auto-discovery does not search the `mtp/` + subfolder — without an explicit `-md `, `--spec-type draft-mtp` + silently runs at the main model's speed. The serve script gates on the + head's presence and passes it. +- **Memory model.** The iGPU's budget is the GTT aperture: 116,732 MiB + (~114 GiB), `gtt_total=122402963456` — not the old "64 GiB rocm-smi" + lesson (stale: the CIRU Orca held 92.6 GiB on the same device). 200k + budget ≈ 100 GiB, ~14 GiB of headroom. +- **Verified before cutover.** Gemma4-12B smoke load on the iGPU via the + prebuilt: clean completion with reasoning extraction (the first probe + returned empty content because the thinking block consumed the 32-token + budget — a budget artifact, not a decode failure). CIRU restored for the + ~3-hour model fetch (health 200 in ~40 s). +- **Docs.** New runbook `frank-large-unsloth-qwen38-flash-next.md`; + `frank-large-ciru-orca.md` marked RETIRED (rollback target, payload and + runtime intact on disk); AGENTS.md topology/section/runbook index and the + GPU-budget lesson updated; LLM index header + mermaid + active list + updated. +- **Files.** `scripts/serve-frank-large.sh`, `scripts/frank-large.env`, + `scripts/frank-large.service`, `scripts/download-frank-large-unsloth.sh`, + `scripts/deploy-remote.sh`, `tests/frank_large_unit.bats`. + ## 2026-09-21 — node-dankclaw Prometheus target fixed (netavark host-bind quirk) - **The gap.** A deep audit found the `node-dankclaw` scrape target red while