diff --git a/AGENTS.md b/AGENTS.md index 7eeb41e..8cfdec5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 | -| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 | +| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 29.3 GiB) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, compiled-in Qwen Jinja, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -227,7 +227,7 @@ one: `ip neigh show ` from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP. 1. Stop and mask (self-healing would restart it): - - Dankclaw `frank-large` (root unit, stock llama.cpp on the iGPU): + - Dankclaw `frank-large` (root unit, gufo on the iGPU): `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` @@ -277,47 +277,79 @@ FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4 projector's audio encoder (frank-small transcribes WAV verbatim even though `audio` is omitted from its `input` list — OMP's schema rejects it). -### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE) +### frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE) -The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151, -`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm -runtime — no host SDK gate) as a **root systemd unit** +**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a +vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 / +Radeon 8060S, `gfx1151`) — as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`), serving -`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 31.5 GiB, -sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, full -262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), DFlash2 spec -decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, depth 4, -`SPEC_DRAFT_N_MAX=4`), reasoning extracted to `reasoning_content`, and -**no vision projector**. It replaced the Kairic Edge ROCmFPX build (jcbtc -Qwen3.8-27B-IU4-Kairic-Edge), which had replaced the unsloth -Qwen3.8-Flash-Next UD-IQ4_XS. +`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 29.3 GiB, +sha256-verified) on port 7770 from the gfx1151 iGPU. One session, full +262,144 ctx (the model's native window, `SESSIONS=1`), DFlash2 spec decode +(the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's **adaptive** block +policy, `--draft-tokens 7`), greedy sampling, reasoning extracted to +`reasoning_content`, and **no vision projector**. It replaced the unsloth +llama.cpp b11030 prebuilt, which had replaced the Kairic Edge ROCmFPX +build. Cut over 2026-09-28. -The spec method is DFlash2 (block-diffusion drafter) at depth 4. The -drafter is loaded via `--model-draft`; the chain is `--spec-type -draft-dflash` with `--spec-draft-device ROCm0`. KV is **q8_0** (the -unsloth b11030 build does not support fp8 KV; q8_0 is the -highest-precision 8-bit option, ~8 GiB at 262k). The serve script -(`scripts/serve-frank-large.sh`) validates binary → main GGUF → drafter -and exits with a pointer message if any is missing, then `exec`s -`llama-server` with `--jinja --alias frank-large --chat-template-kwargs -'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the -Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary -falls back to the built-in `peg-native` template, which produces a -~73,600-token prompt for a ~250-token request). +**The runtime is built from source on the host** — +`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with +`scripts/build-gufo.sh` (`--verify` re-checks an installed tree). gufo ships +no prebuilt Linux binaries and the v0.1.0/v0.1.1 release assets are empty. +Unlike the retired llama.cpp prebuilt, which bundled its own ROCm, **gufo +links the host ROCm 7.2.4 through its RPATH**, so the HIP dev packages must +stay installed. Two host workarounds the build script encodes: -The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR, -`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated" -is not the budget). The 27B dense model (31.5 GiB) is mmap'd in with -`--n-gpu-layers 999`; budget at 262k ≈ 41-43 GiB (weights + KV + buffers), -leaving ~71 GiB of aperture headroom. The 8 GiB prompt-cache budget -(`CACHE_RAM=8192`) is the qualified config. +- **Ubuntu's `librocwmma-dev` must be absent.** It installs a truncated + rocWMMA header set into `/usr/include` that shadows AMD's complete one + and breaks the build with `'internal/accessors.hpp' file not found`. A + `-I` cannot fix it — in HIP mode the host pass searches the system + directories first. The package has no reverse dependencies. +- **`--gcc-install-dir` must name GCC 15.** Ubuntu 26.04 ships + `/usr/include/c++/16`, which the ROCm clang prefers; its `` + needs a `std::__detail::__notify_impl` symbol version the installed + `libstdc++.so.6` (GLIBCXX_3.4.35) does not export, and the link fails. + +`gufo diagnose` is the preflight and must report `[PASS]` with `gfx1151`. +It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for +`serve llm`) after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`, which +matters because ComfyUI holds queues on the same device; four HIP processes +is the device ceiling. + +**The client contract changed in three ways worth knowing.** There is no +`--chat-template-file`: the official Qwen Jinja is compiled in, so the +Qwen-Sharp 73k-token-prompt failure mode cannot recur. `reasoning_effort` +now accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap` +fold for frank-large is no longer *required* — it is kept deliberately, so +that removing it does not silently move every `high` role from `xhigh` to +`high` effort inside a runtime swap. And there is no KV-quantization or +prompt-cache-RAM flag; prompt reuse is on by default and `--cache-disk` +is opt-in and off. + +**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables +(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5× +faster, decode +7 % at zero depth and +40 % at 4 users — but **−14 % at +65,536 tokens of depth and −24 % at 131,072**. On the 8-bit target the +crossover is ~32k of cached context, and frank-large runs one 262,144-token +conversation, so long agent sessions sit on the losing side of that line. +The Q4_K_XL target shows no such regression; revisiting the quantization is +a separate decision from the runtime. Peak memory is 50.58 GiB at the full +262,144 context against the iGPU's 114 GiB GTT aperture. + +The serve script (`scripts/serve-frank-large.sh`) validates binary → main +GGUF → drafter and exits with a pointer message if any is missing, then +`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model +…--served-model-name frank-large --context 262144 --speculative dflash2 …`. +The deployer's `deploy_llm_large` gates on the **runtime binary** as well +as the payload, and returns before it stops anything. Fetch: `scripts/download-frank-large.sh` (per-file, each file verified -against its declared byte count + sha256; `--verify-only` re-checks). +against its declared byte count + sha256; `--verify-only` re-checks). The +payload is unchanged by every frank-large runtime swap since 2026-09-21. -Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md`. -Rollback: [[frank-large-kairic-edge-qwen38-27b]] (payload and runtime -intact on disk). +Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`. +Rollback: [[frank-large-unsloth-qwen38-27b-q8xl]] (the b11030 prebuilt and +the payload are both intact on disk). ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) @@ -771,7 +803,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` | +| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | | frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | @@ -781,6 +813,7 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| +| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` | | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` | | llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` | | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` | diff --git a/README.md b/README.md index 97c0e5c..f8a4c87 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ one deploy script. | Host | User | IP | Podman stack | LLM service | LLM port | |------|------|----|--------------|-------------|----------| | dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 | -| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (CIRU ROCmFPX, Qwen3.8-27B) | 7770 | +| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 | ## Active Services diff --git a/scripts/build-gufo.sh b/scripts/build-gufo.sh new file mode 100755 index 0000000..9771562 --- /dev/null +++ b/scripts/build-gufo.sh @@ -0,0 +1,96 @@ +#!/usr/bin/env bash +# Build the gufo runtime that serves frank-large on Dankclaw. +# +# bash scripts/build-gufo.sh # build + install the pinned tag +# bash scripts/build-gufo.sh --verify # only check the installed tree +# +# gufo (gufo-org/gufo) is the Strix Halo inference engine: it targets +# gfx1151 specifically and has no prebuilt Linux release binaries, so the +# runtime is built from the pinned git tag and installed to +# /home/dank/GitPackages/gufo-/. Everything below is what Dankclaw +# needs on top of AMD's Linux instructions. +# +# The host already carries the ROCm 7.2.4 stack (/opt/rocm -> /etc/ +# alternatives/rocm) with the HIP dev packages, so this script installs +# only the ordinary system libraries gufo links against. +# +# Two host facts this build depends on, both discovered on 2026-09-28: +# +# 1. Ubuntu's librocwmma-dev ships a TRUNCATED rocWMMA header set (the +# public headers without rocwmma/internal/) into /usr/include, where +# it shadows AMD's complete copy under /opt/rocm/include. The first +# build died on "'internal/accessors.hpp' file not found" from +# /usr/include/rocwmma/rocwmma.hpp. A -I cannot fix it: in HIP mode +# the host compilation pass searches the system directories first. +# The package has no reverse dependencies, so it is removed. +# 2. The ROCm clang picks the NEWEST /usr/include/c++ tree, which on +# Ubuntu 26.04 is GCC 16's. Its emits a symbol +# reference the installed libstdc++ (GLIBCXX_3.4.35, GCC 15) does not +# export, and the link fails with "undefined symbol: +# std::__detail::__notify_impl". --gcc-install-dir pins the headers to +# the same GCC the runtime library came from. +set -euo pipefail + +GUFO_TAG="${GUFO_TAG:-v0.1.1}" +GUFO_REPO="${GUFO_REPO:-https://github.com/gufo-org/gufo}" +# The commit the tag resolved to when this runtime was qualified. A tag that +# moves is an unreproducible runtime, so the pin is checked, not assumed. +GUFO_COMMIT="${GUFO_COMMIT:-b0f8673470a3dcc2b5e9c9546c48a40b82b5ff31}" +SRC_DIR="${GUFO_SRC_DIR:-/home/dank/GitPackages/gufo-src}" +PREFIX="${GUFO_PREFIX:-/home/dank/GitPackages/gufo-${GUFO_TAG}}" +JOBS="${JOBS:-$(nproc)}" +# The GCC toolchain that owns the installed libstdc++ on this host. +GCC_DIR="${GCC_DIR:-/usr/lib/gcc/x86_64-linux-gnu/15}" + +# check_pin — refuse a source tree that is not the qualified commit. +check_pin() { + local commit + commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)" + if [[ -n "$commit" && "$commit" != "$GUFO_COMMIT" ]]; then + echo "GUFO_COMMIT mismatch: $SRC_DIR is at $commit, pinned $GUFO_COMMIT" >&2 + exit 1 + fi +} + +if [[ "${1:-}" == "--verify" ]]; then + # The pin first: a moved tag is the more urgent thing to hear about, and + # it is the check that must not be masked by a missing install. + check_pin + [[ -x "$PREFIX/bin/gufo" ]] || { echo "no gufo at $PREFIX/bin/gufo" >&2; exit 1; } + "$PREFIX/bin/gufo" diagnose | tail -n 5 + exit 0 +fi + +echo "== system libraries gufo links against ==" +sudo -n apt-get install -y -qq --no-install-recommends \ + build-essential cmake ninja-build pkg-config \ + libicu-dev libcurl4-openssl-dev libssl-dev libpng-dev libjpeg-dev ffmpeg + +echo "== dropping Ubuntu's truncated rocWMMA headers ==" +if dpkg -s librocwmma-dev >/dev/null 2>&1; then + sudo -n apt-get remove -y -qq librocwmma-dev +fi + +echo "== source: $GUFO_REPO @ $GUFO_TAG ==" +if [[ ! -d "$SRC_DIR/.git" ]]; then + mkdir -p "$(dirname "$SRC_DIR")" + git clone --depth 1 --branch "$GUFO_TAG" "$GUFO_REPO" "$SRC_DIR" +fi +git -C "$SRC_DIR" fetch --depth 1 origin "refs/tags/$GUFO_TAG" +git -C "$SRC_DIR" checkout -q --detach "refs/tags/$GUFO_TAG" +echo " $(git -C "$SRC_DIR" rev-parse HEAD)" +check_pin + +echo "== configure + build (release, HIP gfx1151) ==" +cd "$SRC_DIR" +toolchain="--gcc-install-dir=$GCC_DIR" +cmake --preset release \ + -DCMAKE_INSTALL_PREFIX="$PREFIX" \ + -DCMAKE_HIP_FLAGS="$toolchain" \ + -DCMAKE_EXE_LINKER_FLAGS="$toolchain" +cmake --build --preset release --parallel "$JOBS" +cmake --install build/release + +echo "== verify ==" +"$PREFIX/bin/gufo" diagnose +echo "frank-large runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)" diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index f3b8bc0..697b784 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1077,18 +1077,25 @@ deploy_anubis() { wait_container_healthy anubis 60 || return 1 } -# frank-large is the stock llama.cpp prebuilt (unsloth b11030, ROCm -# gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 -# drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU, -# as a ROOT unit since the 2026-09-19 cutover: the runtime is a prebuilt -# binary, so there is no container image to install and no generated unit -# to dodge. relink_unit points /etc/systemd/system/frank-large.service at -# the repo file and boot persistence is that unit's [Install] section. +# frank-large is gufo (gufo-org/gufo), the Strix Halo inference engine built +# from the pinned v0.1.1 tag by scripts/build-gufo.sh, serving +# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter +# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU. It is a +# ROOT unit: the runtime is a native binary linked against the host ROCm, +# so there is no container image to install and no generated unit to dodge. +# relink_unit points /etc/systemd/system/frank-large.service at the repo +# file and boot persistence is that unit's [Install] section. # -# It replaces the Kairic Edge runtime (ciru-ai/ROCmFPX), which had -# replaced the unsloth llama.cpp prebuilt serving -# unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS. The port gate is the same as -# every cutover since 2026-09-16: nothing starts until 7770 is released. +# The runtime gate matters as much as the payload gate: gufo is built from +# source on this host (there is no prebuilt release), so a machine whose +# /home/dank/GitPackages tree was never built has a unit that can never +# become healthy, and a green deploy would leave the endpoint dark. +# +# It replaces the stock llama.cpp prebuilt (unsloth b11030), which had +# replaced the Kairic Edge runtime (ciru-ai/ROCmFPX). The port gate is the +# same as every cutover since 2026-09-16: nothing starts until 7770 is +# released. +LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}" LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf}" LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}" @@ -1126,6 +1133,11 @@ deploy_llm_large() { # Refuse rather than restart-loop. A unit whose runtime or payload is # missing never becomes healthy, and a green deploy would leave the # endpoint dark with nothing in the deploy log to say why. + if [[ ! -x "$LLM_LARGE_RUNTIME" ]]; then + echo "ERROR: gufo runtime absent at $LLM_LARGE_RUNTIME" >&2 + echo "frank-large: build it: bash scripts/build-gufo.sh" >&2 + return 1 + fi local missing=0 f for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT"; do if [[ ! -f "$f" ]]; then @@ -1140,10 +1152,12 @@ deploy_llm_large() { # On a re-deploy the predecessor is THIS unit: stop it before waiting on # the port, or the wait deadlocks on the server it just stopped serving. # Then sweep a stray manual run of the SAME model -- that is how a bench - # harness leaves the port busy. Match the model path, not the binary name: - # frank-small on this host is also a llama-server and must not be killed. - # The pattern never appears in the deploying shell's own command line, so - # -f is safe here (it is not in `bash deploy-remote.sh`). + # harness leaves the port busy, and the pattern matches the gufo command + # line because it carries the model path. Match the model path, not a + # binary name: frank-small on this host serves a different GGUF over + # llama-server and must not be killed. The pattern never appears in the + # deploying shell's own command line, so -f is safe here (it is not in + # `bash deploy-remote.sh`). run_privileged systemctl stop frank-large.service 2>/dev/null || true pkill -f 'Qwen3.8-27B-UD-Q8_K_XL' 2>/dev/null || true # Nothing may start until the predecessor has released the port. On diff --git a/scripts/frank-large.env b/scripts/frank-large.env index c23dfd2..a1245f3 100644 --- a/scripts/frank-large.env +++ b/scripts/frank-large.env @@ -1,6 +1,5 @@ -# frank-large — stock llama.cpp (mainline, the unsloth b11030 prebuilt) -# serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU -# (gfx1151). +# frank-large — gufo (gufo-org/gufo) serving +# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU (gfx1151). # # These are the tunables read by scripts/serve-frank-large.sh. The flag # list is owned by the serve script; this file only sets fleet values. @@ -10,27 +9,28 @@ # exception because the serve script itself consumes it for --host. # The payload (scripts/download-frank-large.sh): the serve script refuses to -# start without the main GGUF and the DFlash2 drafter. +# start without the main GGUF and the DFlash2 drafter. Both files are the +# same ones the llama.cpp deployment used — the cutover changed the RUNTIME, +# not the weights. MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL -# The stock llama.cpp runtime (the unsloth b11030 prebuilt). -RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030 +# The gufo runtime, built from source by scripts/build-gufo.sh. +RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1 -# Fleet endpoint: the same port and alias the prior deployments served, so +# Fleet endpoint: the same port and alias every prior frank-large served, so # no client, probe or models.yml entry changes address. +ALIAS=frank-large HOST=0.0.0.0 PORT=7770 -# One stream, full 262144 context (the model's native window). +# One session, full 262144 context (the model's native window). CONTEXT_SIZE=262144 -PARALLEL_SLOTS=1 +SESSIONS=1 -# DFlash2 spec decode (a block-diffusion drafter) at depth 4. The drafter -# has a sliding window of 2048 (small draft KV, ~40 MiB). -SPEC_DRAFT_N_MAX=4 - -# Prompt-cache budget: 8 GiB. The model is a 27B hybrid (16 full attention -# + 48 Gated DeltaNet linear attention); its DeltaNet state is smaller than -# a primarily-DeltaNet model, so the fleet's 12.8 GiB DeltaNet floor does -# not apply. -CACHE_RAM=8192 +# DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft +# block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is +# 7, chosen over llama.cpp's fixed depth 4 because gufo's adaptive policy +# shortens the block when acceptance drops) and DRAFT_POLICY is the block +# length controller. +DRAFT_TOKENS=7 +DRAFT_POLICY=adaptive diff --git a/scripts/frank-large.service b/scripts/frank-large.service index df23ca4..7c9e9a4 100644 --- a/scripts/frank-large.service +++ b/scripts/frank-large.service @@ -1,5 +1,5 @@ [Unit] -Description=stock llama.cpp (unsloth b11030, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, fp8 KV, 262144 ctx / 1 slot, root systemd unit (dankclaw, frank-large, Strix Halo iGPU) +Description=gufo (gufo-org/gufo, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU) After=network-online.target Wants=network-online.target diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 6daf828..2e4005d 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -1,32 +1,36 @@ #!/usr/bin/env bash -# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on -# Dankclaw's Strix Halo iGPU (gfx1151), using stock llama.cpp -# (mainline, the unsloth b11030 prebuilt). +# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on Dankclaw's +# Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the inference +# engine built specifically for this hardware. # -# Runtime: /home/dank/GitPackages/unsloth-llama-b11030/llama-server -# (mainline, ROCm gfx1151 prebuilt). The build supports the qwen3_5 -# architecture and the DFlash2 drafter (llama_model_qwen35, -# llama_model_dflash, common_speculative_impl_draft_dflash). +# Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source +# against the host ROCm by scripts/build-gufo.sh. It replaces the stock +# llama.cpp prebuilt (unsloth b11030) that served this endpoint from +# 2026-09-21; payload, port and alias are unchanged, so no client moved. # -# Shape: one stream, full 262144 context (the model's native window), all -# layers on the iGPU (--n-gpu-layers 999). The 27B dense model (31.46 GiB GGUF) + the -# DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) + compute buffers fits the -# iGPU's GTT aperture (~114 GiB). +# Shape: one session, full 262144 context (the model's native window). The +# 27B dense weights (29.30 GiB) + the DFlash2 drafter (1.07 GiB) + KV + +# compute buffers fit the iGPU's GTT aperture (~114 GiB); gufo's own +# Q8/AR table measures 50.58 GiB peak at this context capacity. # -# Spec method: DFlash2 (a block-diffusion drafter) at depth 4. The drafter -# has a sliding window of 2048 (small draft KV, ~40 MiB). +# Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block +# policy, capped at DRAFT_TOKENS proposals per step. The drafter is the +# z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048 +# (small draft KV). # -# KV: q8_0 quantized (--cache-type-k q8_0 --cache-type-v q8_0). The unsloth b11030 -# build does not support fp8 KV (allowed: f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/ -# q5_1); q8_0 is the highest-precision 8-bit option and preserves the ~8 GiB footprint. +# Sampling: greedy (--temperature 0). That is what this endpoint has always +# served, and it keeps the swap to a pure runtime change; gufo's per-model +# recommendation for Qwen3.8 thinking is sampled (1.0 / 0.95 / top-k 20) and +# is a separate decision. # -# Text-only (no mmproj) — matching the prior frank-large. +# Template: gufo compiles the official Qwen Jinja into the binary and takes +# no --chat-template-file. Its Qwen27B default is thinking ON at xhigh effort +# with prior reasoning preserved, which is the contract the fleet already +# depends on, so --think auto + --preserve-thinking on is spelled out here +# rather than left implicit. # -# The Qwen3.8 family template keeps prior thinking blocks across turns when -# preserve_thinking is set; pass --jinja and the kwarg so multi-turn agents -# do not lose the trail. The template gates reasoning_effort to {xhigh, -# medium, low} — the harness folds its levels onto that set in models.yml -# (reasoningEffortMap). +# Text-only: gufo auto-discovers a mmproj beside the model and this payload +# has none, so the endpoint stays text-only. # # Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides # it; anything not set there falls through to the defaults below. @@ -41,22 +45,23 @@ if [[ -f "$ENV_FILE" ]]; then fi # --- defaults (used when the env file does not set them) ------------------ -: "${RUNTIME_DIR:=/home/dank/GitPackages/unsloth-llama-b11030}" +: "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}" : "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}" +: "${ALIAS:=frank-large}" : "${HOST:=0.0.0.0}" : "${PORT:=7770}" : "${CONTEXT_SIZE:=262144}" -: "${PARALLEL_SLOTS:=1}" -: "${SPEC_DRAFT_N_MAX:=4}" -: "${CACHE_RAM:=8192}" -TEMPLATE_FILE="${TEMPLATE_FILE:-$SCRIPT_DIR/qwen-sharp-v22.4.0.jinja}" +: "${SESSIONS:=1}" +: "${DRAFT_TOKENS:=7}" +: "${DRAFT_POLICY:=adaptive}" -SERVER_BIN="$RUNTIME_DIR/llama-server" +GUFO_BIN="$RUNTIME_DIR/bin/gufo" MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf" DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" -if [[ ! -x "$SERVER_BIN" ]]; then - echo "runtime not found at $SERVER_BIN" >&2 +if [[ ! -x "$GUFO_BIN" ]]; then + echo "runtime not found at $GUFO_BIN" >&2 + echo "frank-large: build it: bash scripts/build-gufo.sh" >&2 exit 1 fi for f in "$MODEL_FILE" "$DRAFTER_FILE"; do @@ -66,42 +71,28 @@ for f in "$MODEL_FILE" "$DRAFTER_FILE"; do fi done -# --- env (the unsloth build's ROCm setup) -------------------------------- +# --- env ------------------------------------------------------------------- +# gufo links against the host ROCm through its RPATH (/opt/rocm -> +# /etc/alternatives/rocm), so no LD_LIBRARY_PATH is needed. The device is +# already gfx1151 — `gufo diagnose` reports it natively — so no +# HSA_OVERRIDE_GFX_VERSION either; the override existed for the prebuilt +# llama.cpp runtime and can only misreport the arch here. export HIP_VISIBLE_DEVICES=0 -export HSA_OVERRIDE_GFX_VERSION=11.5.1 -export LD_LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/llvm/lib:${LD_LIBRARY_PATH:-}" # --- the serve command --------------------------------------------------- -ARGS=( - --host "$HOST" - --port "$PORT" - --model "$MODEL_FILE" - --alias frank-large - --ctx-size "$CONTEXT_SIZE" - --parallel "$PARALLEL_SLOTS" - --n-gpu-layers 999 - --jinja - --metrics - --chat-template-file "$TEMPLATE_FILE" - --chat-template-kwargs '{"preserve_thinking":true}' - --cache-type-k q8_0 - --cache-type-v q8_0 - --cache-ram "$CACHE_RAM" - # DFlash2 spec decode (a block-diffusion drafter) - --spec-type draft-dflash - --model-draft "$DRAFTER_FILE" - --spec-draft-n-max "$SPEC_DRAFT_N_MAX" - --spec-draft-n-min 0 - --spec-draft-device ROCm0 - # Greedy sampling (the DFlash2 drafter needs greedy verification) - --temp 0 - --top-p 1.0 - --top-k 0 - --min-p 0.0 - # Reasoning (thoughts land in reasoning_content) - --reasoning auto - --reasoning-format deepseek - --reasoning-budget -1 -) - -exec "$SERVER_BIN" "${ARGS[@]}" +# Server options precede the subcommand, model options follow it. +exec "$GUFO_BIN" serve \ + --host "$HOST" \ + --port "$PORT" \ + --sessions "$SESSIONS" \ + llm \ + --model "$MODEL_FILE" \ + --served-model-name "$ALIAS" \ + --context "$CONTEXT_SIZE" \ + --speculative dflash2 \ + --dflash-model "$DRAFTER_FILE" \ + --draft-tokens "$DRAFT_TOKENS" \ + --draft-policy "$DRAFT_POLICY" \ + --temperature 0 \ + --think auto \ + --preserve-thinking on diff --git a/serverAppConfigs-docs/LLM/Dankclaw System Specs.md b/serverAppConfigs-docs/LLM/Dankclaw System Specs.md index ef35802..6cce340 100644 --- a/serverAppConfigs-docs/LLM/Dankclaw System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankclaw System Specs.md @@ -101,8 +101,8 @@ cable between claw and server is required for 10G. - Unified memory eliminates separate VRAM limits. The GPU uses system RAM. - The GPU budget is the GTT aperture: ~114 GiB (`gtt_total` in `/sys/class/drm/card1/device/`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget. -- The active service is `frank-large` on port 7770: stock llama.cpp (the unsloth b11030 prebuilt) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, q8_0 KV, DFlash2 drafter). See [[frank-large-unsloth-qwen38-27b-q8xl]]. -- The active runtime (the unsloth b11030 prebuilt) **bundles its own ROCm runtime** next to the binary (`RPATH=$ORIGIN/../lib`), so it has no host SDK gate — the system ROCm 7.2.4 is not on its path. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is no longer in the serving path. +- The active service is `frank-large` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-large-gufo-qwen38-27b]]. +- The active runtime **links against the host ROCm** (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH, so it needs the host HIP stack rather than a bundled one — the opposite of the retired llama.cpp prebuilt, which carried its own. Building it needs the ROCm dev packages that are installed, and two workarounds recorded in the runbook (drop `librocwmma-dev`; pin `--gcc-install-dir` to GCC 15). The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is not in the serving path. - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18. - 1.1 TB free disk space is ample for model storage. diff --git a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md new file mode 100644 index 0000000..aecbdb9 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md @@ -0,0 +1,176 @@ +--- +type: reference +title: "gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)" +description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the one-session / full-window shape, what changed in the client contract versus llama.cpp, the measured prefill/decode trade, and the failure modes worth knowing." +tags: + - llm + - gufo + - unsloth + - dankclaw + - frank-large + - strix-halo + - gfx1151 + - q8 + - dflash2 + - rocm +timestamp: 2026-09-28 +--- + +# frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) + +> **ACTIVE since 2026-09-28.** Replaces +> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth +> b11030 prebuilt), which had replaced +> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same +> alias (`frank-large`), **same weights**. + +## Identity + +- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw). +- **Runtime:** [gufo](https://github.com/gufo-org/gufo) — "the Strix Halo + inference engine", a vertical engine built for exactly this hardware + (Ryzen AI MAX+ 395 / Radeon 8060S, `gfx1151`, unified memory). Pinned at + **v0.1.1** (commit `b0f8673`), built from source on the host to + `/home/dank/GitPackages/gufo-v0.1.1/`. +- **Service shape:** root systemd unit `scripts/frank-large.service` + (`User=dank`, `LimitMEMLOCK=infinity`), launched by + `scripts/serve-frank-large.sh` with tunables from + `scripts/frank-large.env`. + +gufo ships **no prebuilt Linux binaries** (the v0.1.0/v0.1.1 release assets +are empty) and no Nix on this host, so the runtime is a source build. It is +also young: v0.1.0 landed 2026-09-28, the same day as this cutover. + +## Why this engine + +gufo's own measurement of the two engines on the same Qwen3.8-27B targets +(Strix Halo, `628ed18e` vs llama.cpp `b11069`, greedy, DFlash2): + +| Q8_K_XL, DFlash2 | gufo | llama.cpp | delta | +|---|---:|---:|---:| +| prefill, depth 0 | 457.05 t/s | 333.85 t/s | **+36.9 %** | +| prefill, depth 131,072 | 237.24 t/s | 131.80 t/s | **+80.0 %** | +| decode, depth 0 (mixed) | 16.08 t/s | 15.00 t/s | +7.2 % | +| decode, depth 32,768 (mixed) | 14.41 t/s | 14.44 t/s | −0.2 % | +| decode, depth 65,536 (mixed) | 10.87 t/s | 12.62 t/s | **−13.9 %** | +| decode, depth 131,072 (mixed) | 8.87 t/s | 11.62 t/s | **−23.7 %** | +| cold load to HTTP-ready | 8.17 s | 36.47 s | **4.5× faster** | +| peak memory @ 262,144 ctx | 50.58 GiB | 48.60 GiB | +4 % | + +Read that honestly: **prefill, load time and short-context decode improve; +deep-context decode on the 8-bit target gets worse.** The crossover on Q8 is +around 32k of cached depth. frank-large runs one 262,144-token conversation, +so a long agent session spends its life on the losing side of that line +while a short turn wins. The Q4_K_XL target does not show the regression +(gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to +revisit the quantization separately from the runtime. + +## Build + +`scripts/build-gufo.sh` reproduces the runtime end to end. It clones the +pinned tag, configures the `release` preset (HIP, `--offload-arch=gfx1151`), +builds, installs, and finishes with `gufo diagnose`. Two host facts it +encodes, both found the hard way on 2026-09-28: + +- **Ubuntu's `librocwmma-dev` must not be installed.** It ships a truncated + rocWMMA header set (the public headers without `rocwmma/internal/`) into + `/usr/include`, where it shadows AMD's complete copy under + `/opt/rocm/include`. The first build died on `'internal/accessors.hpp' + file not found` from `/usr/include/rocwmma/rocwmma.hpp`, and **a `-I` + cannot fix it**: in HIP mode the host compilation pass searches the system + directories first. The package has no reverse dependencies; the script + removes it. +- **`--gcc-install-dir` must point at GCC 15.** Ubuntu 26.04 has + `/usr/include/c++/16`, and the ROCm clang picks the newest tree, whose + `` references `std::__detail::__notify_impl` at a symbol + version the installed `libstdc++.so.6` (GLIBCXX_3.4.35) does not export. + The link then fails with `undefined symbol`. Pinning the headers to the + GCC the runtime library came from fixes it. + +Verify an installed runtime with `bash scripts/build-gufo.sh --verify`, or +`gufo diagnose` directly — it reports `[PASS]` plus the detected +architecture, driver and queue budget. + +## Shape + +- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — 27B dense, + 29.30 GiB. **Unchanged by the cutover**, still fetched and sha256-verified + by `scripts/download-frank-large.sh`. +- **Context:** one session, full 262,144 tokens (the model's native window), + `SESSIONS=1`. The fleet's single stream is unchanged. +- **Spec method:** DFlash2, gufo's **adaptive** block policy + (`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the same + z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB). +- **Sampling:** greedy (`--temperature 0`), as every prior frank-large + served. gufo's per-model recommendation for Qwen3.8 thinking is *sampled* + (1.0 / 0.95 / top-k 20) — that is a separate decision, deliberately not + bundled into a runtime swap. +- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak + with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture. + +## Client contract — what actually changed + +Everything the fleet depended on still holds: port 7770, alias `frank-large`, +`reasoning_content`, `tool_calls`, streaming SSE, and text-only (gufo +auto-discovers a `mmproj` beside the model; this payload has none). + +The differences worth knowing: + +- **No custom chat template.** gufo compiles the official Qwen Jinja into the + binary and takes no `--chat-template-file`. The Qwen-Sharp v22.4.0 file + this endpoint used with llama.cpp is **no longer in the command line** — + the 2026-09-21 "73,600-token prompt" incident cannot recur, because there + is no fallback template to land in. Thinking defaults to on at `xhigh` + with prior reasoning preserved (`--think auto --preserve-thinking on`, + spelled out in the serve script). +- **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`, + `low`, `medium`, `high`, `xhigh`, `max`). Qwen-Sharp rejected anything + outside `{xhigh, medium, low}` with an HTTP 500, which is why `models.yml` + still carries a `reasoningEffortMap` fold for this model. **The fold is + kept on purpose**: removing it would silently move every `high` role from + `xhigh` to `high` effort in the same commit as the runtime swap. Drop it as + its own change once the effort curve is a decision rather than a side + effect. +- **No KV-cache quantization flag.** gufo picks its own KV dtype; there is + no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it. + Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in + and currently off. +- **Health/readiness split:** `GET /health` is process liveness and is what + the deployer gates on; `GET /ready` is model/backend readiness. + `GET /metrics` is Prometheus-format. +- **Long bodies:** `--max-request-bytes` defaults to 8 MiB and unknown + request fields are rejected rather than ignored. + +## Failure modes worth knowing + +- **A missing runtime is a deploy error, not a restart loop.** The handler + gates on `LLM_LARGE_RUNTIME` *before* it stops anything, and the serve + script exits with a `build-gufo.sh` pointer rather than exec'ing. +- **Host memory is the ceiling, and it is shared.** frank-large and ComfyUI + both hold gfx1151 compute queues. gufo reads + `/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself + (`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device + time-slices instead of failing; four HIP processes is the device ceiling. +- **Load is seconds, not minutes** (8 s measured cold), so a redeploy no + longer spends 20 minutes in `LLM_WAIT`. +- **A v0.1.x runtime is a moving target.** Rebuilds are not automatic — + `RUNTIME_DIR` names a version, and the deployer will not silently + substitute another. + +## Rollback + +[[frank-large-unsloth-qwen38-27b-q8xl]] — the unsloth b11030 prebuilt is +still at `/home/dank/GitPackages/unsloth-llama-b11030/` and the payload is +the same directory the current runtime reads. Roll back by reverting +`RUNTIME_DIR` (and the gufo-only tunables) in `scripts/frank-large.env`, +restoring the serve script's llama.cpp command, and redeploying. + +# Citations + +- Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT) +- Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo +- API contract: `docs/SERVER.md`; flags: `docs/CLI.md` +- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`; + tunables: `scripts/frank-large.env` +- Payload: `scripts/download-frank-large.sh` +- Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]] diff --git a/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md b/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md index 260cc0c..34c9a2d 100644 --- a/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md +++ b/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md @@ -1,7 +1,7 @@ --- type: reference title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)" -description: "The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing." +description: "The RETIRED frank-large runbook (2026-09-21 → 2026-09-28; replaced by [[frank-large-gufo-qwen38-27b]], and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing." tags: - llm - llama.cpp @@ -19,10 +19,12 @@ timestamp: 2026-09-21 # frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) -> **ACTIVE since 2026-09-21.** Replaces [[frank-large-kairic-edge-qwen38-27b]] -> (Kairic Edge IU4 W4A4, now the rollback target) and -> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it preceded). -> Same port (7770), same alias (`frank-large`), same client contract surface. +> **RETIRED 2026-09-28**, replaced by [[frank-large-gufo-qwen38-27b]] (the +> gufo engine, same weights, same port and alias); the prebuilt is intact on +> disk as the rollback target. ACTIVE from 2026-09-21, when it replaced +> [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge IU4 W4A4) and +> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it +> preceded). ## Identity diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index aaf352f..23a1f6e 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -139,10 +139,18 @@ frank-small, port 7771] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-large-unsloth-qwen38-27b-q8xl]] — ACTIVE frank-large runbook - since 2026-09-21 (root systemd unit, port 7770): stock llama.cpp - (unsloth b11030, ROCm gfx1151) serving +- [[frank-large-gufo-qwen38-27b]] — ACTIVE frank-large runbook since + 2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the + Strix Halo inference engine, built from source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2 + Q4_K_M drafter (adaptive block policy) on the gfx1151 iGPU — one full + 262,144-token conversation, greedy, text-only, the two from-source build + workarounds, the client-contract delta, the measured prefill/decode trade + against llama.cpp, troubleshooting +- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by + [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp + (unsloth b11030, ROCm gfx1151) serving + unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the 114 GiB GTT memory model, client contract, diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 1ea000a..d0e35c5 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -190,13 +190,21 @@ The LLM inference fleet on the two hosts. Full index: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record) -- [[frank-large-unsloth-qwen38-27b-q8xl]] — the ACTIVE frank-large runbook - since 2026-09-21 (root systemd unit, port 7770): stock llama.cpp +- [[frank-large-gufo-qwen38-27b]] — the ACTIVE frank-large runbook since + 2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, + the Strix Halo inference engine, built from source on Dankclaw and + serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the + z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — one 262,144-token + conversation, adaptive block policy, greedy, text-only, the two + from-source build workarounds, what changed in the client contract, the + measured prefill/decode trade against llama.cpp +- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by + [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp (unsloth b11030, ROCm gfx1151) serving - unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the z-lab - DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one 262,144-token - conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp v22.4.0 - template, the 114 GiB GTT memory model, client contract + unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter + (depth 4) on the gfx1151 iGPU — one 262,144-token conversation, q8_0 KV + (the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the + 114 GiB GTT memory model, client contract - [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]; same-day swap): the Kairic Edge runtime (ciru-ai/ROCmFPX kairic-edge-qwen38-27b-v1.2, TheRock 7.15.0, diff --git a/tests/frank_large_unit.bats b/tests/frank_large_unit.bats index e64a927..0f40820 100755 --- a/tests/frank_large_unit.bats +++ b/tests/frank_large_unit.bats @@ -1,31 +1,84 @@ #!/usr/bin/env bats # Seam: the frank-large root unit on dankclaw. # -# The 2026-09-21 cutover moved this service from the Kairic Edge runtime -# (ciru-ai/ROCmFPX) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge onto the -# stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving -# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter -# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M), still as a ROOT unit: a -# prebuilt runtime has no container image to install and no generated -# unit to dodge. This seam pins what a flag list cannot show: the -# operator's one-stream / full-window shape, the payload the serve script -# refuses to start without, the DFlash2 spec wiring, fp8 KV, greedy -# sampling, a deploy handler that gates on the port, and the fact that -# the sweep for a stray benchmark run cannot take frank-small down with -# it. - +# Since 2026-09-28 this endpoint is served by gufo (gufo-org/gufo), the +# Strix Halo inference engine, built from the pinned v0.1.1 tag and serving +# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter. It replaced +# the stock llama.cpp prebuilt (unsloth b11030) on the same port, the same +# alias and the same weights. +# +# Most of this seam runs the serve script against a stub engine and asserts +# on the argv it actually execs, so what is pinned is the command the +# operator gets rather than the text of the script: a tunable renamed, +# dropped, or silently defaulted away fails here the way it would fail a +# deployment. load helpers ENV_FILE="$REPO_ROOT/scripts/frank-large.env" UNIT="$REPO_ROOT/scripts/frank-large.service" SERVE="$REPO_ROOT/scripts/serve-frank-large.sh" FETCH="$REPO_ROOT/scripts/download-frank-large.sh" +BUILD="$REPO_ROOT/scripts/build-gufo.sh" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" -@test "the frank-large unit, env and serve files exist where the deployer installs them from" { +setup() { + STAGE="$(mktemp -d)" + RUNTIME_DIR="$STAGE/gufo" + MODEL_DIR="$STAGE/models" + mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR" + # A stub engine: the serve script execs it, so $@ is exactly what gufo + # would be invoked with. + cat > "$RUNTIME_DIR/bin/gufo" <<'STUB' +#!/usr/bin/env bash +printf '%s\n' "$@" +STUB + chmod +x "$RUNTIME_DIR/bin/gufo" + : > "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf" + : > "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" + STUB_ENV="$STAGE/stub.env" + cat > "$STUB_ENV" < — exact token match, so --port never satisfies a test +# looking for a different flag. +argv_has() { + grep -qxF -- "$1" <<<"$output" +} + +# argv_value — the token following , empty if the flag is absent. +argv_value() { + local i + i="$(grep -nx -- "$1" <<<"$output" | cut -d: -f1)" + [ -n "$i" ] || return 0 + sed -n "$((i + 1))p" <<<"$output" +} + +@test "the frank-large unit, env, serve and build files exist where the deployer installs them from" { [ -f "$ENV_FILE" ] [ -f "$UNIT" ] [ -x "$SERVE" ] + [ -x "$BUILD" ] } @test "frank-large is boot-persistent, self-healing and unprivileged" { @@ -33,65 +86,108 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" grep -q '^Restart=on-failure$' "$UNIT" grep -q '^RestartSec=' "$UNIT" grep -q '^User=dank$' "$UNIT" - # The engine pins staging buffers; the memlock budget must be unbounded - # or the load fails on the RLIMIT. + # The engine pins buffers; the memlock budget must be unbounded or the + # load fails on the RLIMIT. grep -q '^LimitMEMLOCK=infinity$' "$UNIT" # The unit runs the repo's own serve script with the repo's env. grep -q 'ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh' "$UNIT" grep -q 'FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env' "$UNIT" } -@test "the operator's shape is one stream of the full 262144 window with DFlash2 depth 4" { - local ctx slots - ctx="$(sed -nE 's/^CONTEXT_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" - slots="$(sed -nE 's/^PARALLEL_SLOTS=([0-9]+)$/\1/p' "$ENV_FILE")" +@test "the fleet endpoint and alias survive the cutover" { + serve_argv + [ "$status" -eq 0 ] + # frank-large at :7770, whatever engine answers it. + [ "$(argv_value --served-model-name)" = "frank-large" ] + [ "$(argv_value --port)" = "7770" ] + [ "$(argv_value --host)" = "0.0.0.0" ] +} + +@test "the built-in defaults are one session of the full 262144 window with DFlash2" { + serve_argv + [ "$status" -eq 0 ] # 262144 is the model's native window (the operator's call, 2026-09-21). - [ "$ctx" -eq 262144 ] - # One stream. - [ "$slots" -eq 1 ] - # DFlash2 depth 4 (the drafter's block-diffusion depth). - grep -q '^SPEC_DRAFT_N_MAX=4$' "$ENV_FILE" + [ "$(argv_value --context)" -eq 262144 ] + # One session is the single stream every prior frank-large served. + [ "$(argv_value --sessions)" -eq 1 ] + [ "$(argv_value --speculative)" = "dflash2" ] + # A draft ceiling of 0 would silently serve the autoregressive rate. + [ "$(argv_value --draft-tokens)" -gt 0 ] +} + +@test "the shipped env file, not just the defaults, serves the deployed shape" { + # The defaults are a floor; systemd sources the env file. Only the two + # on-host paths are redirected at the stub, so every fleet value in the + # shipped file is the one under test. + sed -e "s#^RUNTIME_DIR=.*#RUNTIME_DIR=$RUNTIME_DIR#" \ + -e "s#^MODEL_DIR=.*#MODEL_DIR=$MODEL_DIR#" \ + "$ENV_FILE" > "$STAGE/shipped.env" + serve_argv "$STAGE/shipped.env" + [ "$status" -eq 0 ] + [ "$(argv_value --served-model-name)" = "frank-large" ] + [ "$(argv_value --host)" = "0.0.0.0" ] + [ "$(argv_value --port)" = "7770" ] + [ "$(argv_value --context)" -eq 262144 ] + [ "$(argv_value --sessions)" -eq 1 ] + [ "$(argv_value --speculative)" = "dflash2" ] + [ "$(argv_value --draft-tokens)" -gt 0 ] + [ "$(argv_value --draft-policy)" = "adaptive" ] +} + +@test "the engine is pointed at the payload that is actually on disk" { + serve_argv + [ "$status" -eq 0 ] + [ -f "$(argv_value --model)" ] + # A drafter path that does not exist would load the target and decode at + # the autoregressive rate, which reads as "the new runtime is slow". + [ -f "$(argv_value --dflash-model)" ] + [ "$(argv_value --model)" != "$(argv_value --dflash-model)" ] } @test "frank-large stays text-only" { - # The new model is text-only (no vision projector). The fleet's - # image path is frank-small; models.yml declares frank-large text-only. - ! grep -q -- '--mmproj' "$SERVE" + # The payload has no mmproj, so an --mmproj would point at nothing. The + # fleet's image path is frank-small; models.yml declares frank-large + # text-only. + serve_argv + ! argv_has --mmproj grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh" } -@test "the serve script pins the stock llama.cpp runtime and refuses a missing binary" { - grep -q 'unsloth-llama-b11030' "$SERVE" - grep -q 'llama-server' "$SERVE" - # A missing runtime or payload is a deploy error with a pointer, not a - # restart loop. - grep -q 'runtime not found' "$SERVE" - grep -q 'missing asset' "$SERVE" - # The Qwen3.8 template keeps prior thinking blocks across turns only - # when the kwarg is set. - grep -q 'preserve_thinking' "$SERVE" - # The chat template file is wired (not just the kwarg) — the 73k-token - # fallback incident (2026-09-21) was a missing --chat-template-file. - grep -q 'chat-template-file' "$SERVE" +@test "the engine is asked for greedy decoding and for preserved prior reasoning" { + # Greedy is what this endpoint has always served; a silent switch to + # sampled output would be a model-behaviour change disguised as a + # runtime swap. Prior thinking blocks must survive across turns or a + # multi-turn agent loses its own reasoning between tool calls. + serve_argv + [ "$(argv_value --temperature)" = "0" ] + [ "$(argv_value --preserve-thinking)" = "on" ] } -@test "the serve script wires the DFlash2 spec method and the drafter" { - # The new model uses DFlash2 (block-diffusion drafter) at depth 4. - grep -q -- '--spec-type draft-dflash' "$SERVE" - grep -q -- '--model-draft' "$SERVE" - grep -q -- '--spec-draft-n-max' "$SERVE" - # The drafter runs on the iGPU (ROCm0). - grep -q -- '--spec-draft-device ROCm0' "$SERVE" +@test "the serve script refuses a missing runtime with the build pointer" { + rm -f "$RUNTIME_DIR/bin/gufo" + serve_argv + [ "$status" -ne 0 ] + [[ "$output" == *"runtime not found"* ]] + [[ "$output" == *"build-gufo.sh"* ]] + # A restart loop would be the alternative; nothing may be exec'd. + ! argv_has serve } -@test "the serve script uses q8_0 KV and greedy sampling" { - # q8_0 KV — the unsloth b11030 build does not support fp8 KV (allowed: - # f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the - # highest-precision 8-bit option and preserves the ~8 GiB footprint. - grep -q -- '--cache-type-k q8_0' "$SERVE" - grep -q -- '--cache-type-v q8_0' "$SERVE" - # Greedy sampling (DFlash2 needs greedy verification). - grep -q -- '--temp 0' "$SERVE" +@test "the serve script refuses a missing drafter with the fetch pointer" { + rm -f "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" + serve_argv + [ "$status" -ne 0 ] + [[ "$output" == *"missing asset"* ]] + [[ "$output" == *"download-frank-large.sh"* ]] + ! argv_has serve +} + +@test "the serve script refuses a missing target and names the file" { + rm -f "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf" + serve_argv + [ "$status" -ne 0 ] + [[ "$output" == *"missing asset"* ]] + [[ "$output" == *"Qwen3.8-27B-UD-Q8_K_XL.gguf"* ]] } @test "the payload is fetched resumably and verified against declared sizes + sha256" { @@ -103,41 +199,72 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" grep -q 'SHA256 MISMATCH' "$FETCH" } -@test "the fleet endpoint and alias survive the cutover" { - grep -q '^HOST=0.0.0.0$' "$ENV_FILE" - grep -q '^PORT=7770$' "$ENV_FILE" - grep -q '^MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL$' "$ENV_FILE" - grep -q -- '--alias frank-large' "$SERVE" +@test "the build script refuses a source tree that is not the pinned commit" { + # gufo ships no prebuilt Linux binaries, so the runtime is built here. + # A tag that moves under us would silently install different kernels + # under the same RUNTIME_DIR, so the pin is enforced, not documented. + local src="$STAGE/not-the-pinned-commit" + mkdir -p "$src" + git -C "$src" init -q + git -C "$src" -c user.email=t@t -c user.name=t commit -q --allow-empty -m drift + run env GUFO_SRC_DIR="$src" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify + [ "$status" -ne 0 ] + [[ "$output" == *"GUFO_COMMIT mismatch"* ]] +} + +@test "the build script reports a runtime that was never built" { + # --verify on a host with no install must say so rather than exit 0 on + # an empty check. + run env GUFO_SRC_DIR="$STAGE/no-source-tree" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify + [ "$status" -ne 0 ] + [[ "$output" == *"no gufo at"* ]] +} + +@test "the deploy handler gates on the runtime, the payload and the port" { + grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY" + grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY" + grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY" + grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY" + grep -q 'LLM_LARGE_RUNTIME' "$REMOTE_DEPLOY" + grep -q 'LLM_LARGE_MODEL' "$REMOTE_DEPLOY" + grep -q 'LLM_LARGE_DRAFT' "$REMOTE_DEPLOY" +} + +@test "the deploy handler will not stop a serving endpoint it cannot replace" { + # The runtime gate must be live (a real conditional on an executable + # path, not a branch that never fires) and it must return BEFORE the + # stop, or a host whose gufo was never built takes frank-large down on + # its way to reporting the failure. + local large gate bail stop + large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")" + [[ "$large" == *'build-gufo.sh'* ]] + gate="$(grep -n 'if \[\[ ! -x "\$LLM_LARGE_RUNTIME" \]\]' <<<"$large" | head -1 | cut -d: -f1)" + stop="$(grep -n 'systemctl stop frank-large' <<<"$large" | head -1 | cut -d: -f1)" + [ -n "$gate" ] + [ -n "$stop" ] + bail="$(awk -v g="$gate" 'NR > g && /return 1/ { print NR; exit }' <<<"$large")" + [ -n "$bail" ] + [ "$bail" -lt "$stop" ] +} + +@test "the stray-run sweep cannot take frank-small down" { + # frank-small on this host serves a different GGUF. The sweep must match + # the model path, not a binary name, or a frank-large deploy restarts the + # 3090 endpoint. + grep -q "pkill -f 'Qwen3.8-27B-UD-Q8_K_XL'" "$REMOTE_DEPLOY" + local large + large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")" + ! grep -q 'pkill -x llama-server' <<<"$large" + ! grep -q 'pkill -x gufo' <<<"$large" } @test "the deploy handler retires the halogen quadlet without deleting it" { grep -q 'retire_halogen_large_quadlet' "$REMOTE_DEPLOY" grep -q 'HOME/.AGENTTRASH' "$REMOTE_DEPLOY" ! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?.*frank-large' "$REMOTE_DEPLOY" -} - -@test "the deploy handler installs the root unit, gates the port and refuses a missing payload" { - grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY" - grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY" - grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY" - grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY" - grep -q 'LLM_LARGE_MODEL' "$REMOTE_DEPLOY" - grep -q 'LLM_LARGE_DRAFT' "$REMOTE_DEPLOY" -} - -@test "the stray-run sweep cannot take frank-small down" { - # frank-small on this host is ALSO a llama-server. The sweep must match - # the model path, not the binary name, or a frank-large deploy restarts - # the 3090 endpoint. - grep -q "pkill -f 'Qwen3.8-27B-UD-Q8_K_XL'" "$REMOTE_DEPLOY" - # ...and only inside the large handler: extract it and check there is no - # bare binary-name pkill in it. - local large - large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")" - ! grep -q 'pkill -x llama-server' <<<"$large" -} - -@test "the halogen quadlet is archived, not live" { + # Nothing live ships for this endpoint as a container any more; the + # archived copy is host state in an ignored .AGENTTRASH, not a tracked + # file. [ ! -e "$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container" ] - [ -f "$REPO_ROOT/dankclaw/.AGENTTRASH/frank-large.container.halogen" ] + ! grep -rq 'Image=.*halogen' "$REPO_ROOT/dankclaw" }