frank-large: replace stock llama.cpp with gufo (gufo-org/gufo) on Dankclaw
gufo is a vertical inference engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large has swapped runtimes seven times in search of a better one. This one ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on the host and the payload, port, alias and client contract stay exactly where they are. Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1, built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4 through its RPATH — the opposite of the unsloth prebuilt it replaces, which carried its own — so two host workarounds are encoded in that script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set that shadows AMD's and cannot be overridden with -I in HIP mode, and the ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose libstdc++ is actually installed. Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL target and the same DFlash2 drafter, greedy, text-only. Speculation moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy capped at 7 proposals. Client-contract deltas, all documented: the chat template is compiled into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token failure mode is gone), reasoning_effort now takes the full OpenAI set (so models.yml's reasoningEffortMap fold is no longer required — kept anyway, so this commit does not also move every high role from xhigh to high effort), and there is no KV-quant or prompt-cache-RAM flag. The test seam now runs the serve script against a stub engine and asserts on the argv it actually execs, instead of grepping the script's text. The deploy handler gates on the runtime binary and returns before it stops the serving unit. Measured trade, from gufo's own Q8_K_XL tables against llama.cpp b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the 8-bit target the crossover is ~32k, and frank-large runs one 262,144 conversation. The Q4_K_XL target shows no such regression.
This commit is contained in:
Frank
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
|
||||
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 29.3 GiB) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, compiled-in Qwen Jinja, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
@@ -227,7 +227,7 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
|
||||
merely wedged kernel usually still answers ARP.
|
||||
|
||||
1. Stop and mask (self-healing would restart it):
|
||||
- Dankclaw `frank-large` (root unit, stock llama.cpp on the iGPU):
|
||||
- Dankclaw `frank-large` (root unit, gufo on the iGPU):
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
@@ -277,47 +277,79 @@ FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
|
||||
projector's audio encoder (frank-small transcribes WAV verbatim even though
|
||||
`audio` is omitted from its `input` list — OMP's schema rejects it).
|
||||
|
||||
### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
|
||||
### frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
|
||||
|
||||
The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151,
|
||||
`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm
|
||||
runtime — no host SDK gate) as a **root systemd unit**
|
||||
**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a
|
||||
vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 /
|
||||
Radeon 8060S, `gfx1151`) — as a **root systemd unit**
|
||||
(`scripts/frank-large.service`, `User=dank`), serving
|
||||
`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 31.5 GiB,
|
||||
sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, full
|
||||
262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), DFlash2 spec
|
||||
decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, depth 4,
|
||||
`SPEC_DRAFT_N_MAX=4`), reasoning extracted to `reasoning_content`, and
|
||||
**no vision projector**. It replaced the Kairic Edge ROCmFPX build (jcbtc
|
||||
Qwen3.8-27B-IU4-Kairic-Edge), which had replaced the unsloth
|
||||
Qwen3.8-Flash-Next UD-IQ4_XS.
|
||||
`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 29.3 GiB,
|
||||
sha256-verified) on port 7770 from the gfx1151 iGPU. One session, full
|
||||
262,144 ctx (the model's native window, `SESSIONS=1`), DFlash2 spec decode
|
||||
(the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's **adaptive** block
|
||||
policy, `--draft-tokens 7`), greedy sampling, reasoning extracted to
|
||||
`reasoning_content`, and **no vision projector**. It replaced the unsloth
|
||||
llama.cpp b11030 prebuilt, which had replaced the Kairic Edge ROCmFPX
|
||||
build. Cut over 2026-09-28.
|
||||
|
||||
The spec method is DFlash2 (block-diffusion drafter) at depth 4. The
|
||||
drafter is loaded via `--model-draft`; the chain is `--spec-type
|
||||
draft-dflash` with `--spec-draft-device ROCm0`. KV is **q8_0** (the
|
||||
unsloth b11030 build does not support fp8 KV; q8_0 is the
|
||||
highest-precision 8-bit option, ~8 GiB at 262k). The serve script
|
||||
(`scripts/serve-frank-large.sh`) validates binary → main GGUF → drafter
|
||||
and exits with a pointer message if any is missing, then `exec`s
|
||||
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
|
||||
'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
|
||||
Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
|
||||
falls back to the built-in `peg-native` template, which produces a
|
||||
~73,600-token prompt for a ~250-token request).
|
||||
**The runtime is built from source on the host** —
|
||||
`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with
|
||||
`scripts/build-gufo.sh` (`--verify` re-checks an installed tree). gufo ships
|
||||
no prebuilt Linux binaries and the v0.1.0/v0.1.1 release assets are empty.
|
||||
Unlike the retired llama.cpp prebuilt, which bundled its own ROCm, **gufo
|
||||
links the host ROCm 7.2.4 through its RPATH**, so the HIP dev packages must
|
||||
stay installed. Two host workarounds the build script encodes:
|
||||
|
||||
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
|
||||
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
|
||||
is not the budget). The 27B dense model (31.5 GiB) is mmap'd in with
|
||||
`--n-gpu-layers 999`; budget at 262k ≈ 41-43 GiB (weights + KV + buffers),
|
||||
leaving ~71 GiB of aperture headroom. The 8 GiB prompt-cache budget
|
||||
(`CACHE_RAM=8192`) is the qualified config.
|
||||
- **Ubuntu's `librocwmma-dev` must be absent.** It installs a truncated
|
||||
rocWMMA header set into `/usr/include` that shadows AMD's complete one
|
||||
and breaks the build with `'internal/accessors.hpp' file not found`. A
|
||||
`-I` cannot fix it — in HIP mode the host pass searches the system
|
||||
directories first. The package has no reverse dependencies.
|
||||
- **`--gcc-install-dir` must name GCC 15.** Ubuntu 26.04 ships
|
||||
`/usr/include/c++/16`, which the ROCm clang prefers; its `<atomic_wait.h>`
|
||||
needs a `std::__detail::__notify_impl` symbol version the installed
|
||||
`libstdc++.so.6` (GLIBCXX_3.4.35) does not export, and the link fails.
|
||||
|
||||
`gufo diagnose` is the preflight and must report `[PASS]` with `gfx1151`.
|
||||
It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for
|
||||
`serve llm`) after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`, which
|
||||
matters because ComfyUI holds queues on the same device; four HIP processes
|
||||
is the device ceiling.
|
||||
|
||||
**The client contract changed in three ways worth knowing.** There is no
|
||||
`--chat-template-file`: the official Qwen Jinja is compiled in, so the
|
||||
Qwen-Sharp 73k-token-prompt failure mode cannot recur. `reasoning_effort`
|
||||
now accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap`
|
||||
fold for frank-large is no longer *required* — it is kept deliberately, so
|
||||
that removing it does not silently move every `high` role from `xhigh` to
|
||||
`high` effort inside a runtime swap. And there is no KV-quantization or
|
||||
prompt-cache-RAM flag; prompt reuse is on by default and `--cache-disk`
|
||||
is opt-in and off.
|
||||
|
||||
**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables
|
||||
(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5×
|
||||
faster, decode +7 % at zero depth and +40 % at 4 users — but **−14 % at
|
||||
65,536 tokens of depth and −24 % at 131,072**. On the 8-bit target the
|
||||
crossover is ~32k of cached context, and frank-large runs one 262,144-token
|
||||
conversation, so long agent sessions sit on the losing side of that line.
|
||||
The Q4_K_XL target shows no such regression; revisiting the quantization is
|
||||
a separate decision from the runtime. Peak memory is 50.58 GiB at the full
|
||||
262,144 context against the iGPU's 114 GiB GTT aperture.
|
||||
|
||||
The serve script (`scripts/serve-frank-large.sh`) validates binary → main
|
||||
GGUF → drafter and exits with a pointer message if any is missing, then
|
||||
`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model
|
||||
…--served-model-name frank-large --context 262144 --speculative dflash2 …`.
|
||||
The deployer's `deploy_llm_large` gates on the **runtime binary** as well
|
||||
as the payload, and returns before it stops anything.
|
||||
|
||||
Fetch: `scripts/download-frank-large.sh` (per-file, each file verified
|
||||
against its declared byte count + sha256; `--verify-only` re-checks).
|
||||
against its declared byte count + sha256; `--verify-only` re-checks). The
|
||||
payload is unchanged by every frank-large runtime swap since 2026-09-21.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md`.
|
||||
Rollback: [[frank-large-kairic-edge-qwen38-27b]] (payload and runtime
|
||||
intact on disk).
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`.
|
||||
Rollback: [[frank-large-unsloth-qwen38-27b-q8xl]] (the b11030 prebuilt and
|
||||
the payload are both intact on disk).
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -771,7 +803,7 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` |
|
||||
| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
@@ -781,6 +813,7 @@ Retired (rollback targets and history):
|
||||
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
|
||||
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
|
||||
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
|
||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
|
||||
|
||||
@@ -13,7 +13,7 @@ one deploy script.
|
||||
| Host | User | IP | Podman stack | LLM service | LLM port |
|
||||
|------|------|----|--------------|-------------|----------|
|
||||
| dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 |
|
||||
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (CIRU ROCmFPX, Qwen3.8-27B) | 7770 |
|
||||
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 |
|
||||
|
||||
## Active Services
|
||||
|
||||
|
||||
Executable
+96
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the gufo runtime that serves frank-large on Dankclaw.
|
||||
#
|
||||
# bash scripts/build-gufo.sh # build + install the pinned tag
|
||||
# bash scripts/build-gufo.sh --verify # only check the installed tree
|
||||
#
|
||||
# gufo (gufo-org/gufo) is the Strix Halo inference engine: it targets
|
||||
# gfx1151 specifically and has no prebuilt Linux release binaries, so the
|
||||
# runtime is built from the pinned git tag and installed to
|
||||
# /home/dank/GitPackages/gufo-<tag>/. Everything below is what Dankclaw
|
||||
# needs on top of AMD's Linux instructions.
|
||||
#
|
||||
# The host already carries the ROCm 7.2.4 stack (/opt/rocm -> /etc/
|
||||
# alternatives/rocm) with the HIP dev packages, so this script installs
|
||||
# only the ordinary system libraries gufo links against.
|
||||
#
|
||||
# Two host facts this build depends on, both discovered on 2026-09-28:
|
||||
#
|
||||
# 1. Ubuntu's librocwmma-dev ships a TRUNCATED rocWMMA header set (the
|
||||
# public headers without rocwmma/internal/) into /usr/include, where
|
||||
# it shadows AMD's complete copy under /opt/rocm/include. The first
|
||||
# build died on "'internal/accessors.hpp' file not found" from
|
||||
# /usr/include/rocwmma/rocwmma.hpp. A -I cannot fix it: in HIP mode
|
||||
# the host compilation pass searches the system directories first.
|
||||
# The package has no reverse dependencies, so it is removed.
|
||||
# 2. The ROCm clang picks the NEWEST /usr/include/c++ tree, which on
|
||||
# Ubuntu 26.04 is GCC 16's. Its <atomic_wait.h> emits a symbol
|
||||
# reference the installed libstdc++ (GLIBCXX_3.4.35, GCC 15) does not
|
||||
# export, and the link fails with "undefined symbol:
|
||||
# std::__detail::__notify_impl". --gcc-install-dir pins the headers to
|
||||
# the same GCC the runtime library came from.
|
||||
set -euo pipefail
|
||||
|
||||
GUFO_TAG="${GUFO_TAG:-v0.1.1}"
|
||||
GUFO_REPO="${GUFO_REPO:-https://github.com/gufo-org/gufo}"
|
||||
# The commit the tag resolved to when this runtime was qualified. A tag that
|
||||
# moves is an unreproducible runtime, so the pin is checked, not assumed.
|
||||
GUFO_COMMIT="${GUFO_COMMIT:-b0f8673470a3dcc2b5e9c9546c48a40b82b5ff31}"
|
||||
SRC_DIR="${GUFO_SRC_DIR:-/home/dank/GitPackages/gufo-src}"
|
||||
PREFIX="${GUFO_PREFIX:-/home/dank/GitPackages/gufo-${GUFO_TAG}}"
|
||||
JOBS="${JOBS:-$(nproc)}"
|
||||
# The GCC toolchain that owns the installed libstdc++ on this host.
|
||||
GCC_DIR="${GCC_DIR:-/usr/lib/gcc/x86_64-linux-gnu/15}"
|
||||
|
||||
# check_pin — refuse a source tree that is not the qualified commit.
|
||||
check_pin() {
|
||||
local commit
|
||||
commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)"
|
||||
if [[ -n "$commit" && "$commit" != "$GUFO_COMMIT" ]]; then
|
||||
echo "GUFO_COMMIT mismatch: $SRC_DIR is at $commit, pinned $GUFO_COMMIT" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--verify" ]]; then
|
||||
# The pin first: a moved tag is the more urgent thing to hear about, and
|
||||
# it is the check that must not be masked by a missing install.
|
||||
check_pin
|
||||
[[ -x "$PREFIX/bin/gufo" ]] || { echo "no gufo at $PREFIX/bin/gufo" >&2; exit 1; }
|
||||
"$PREFIX/bin/gufo" diagnose | tail -n 5
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "== system libraries gufo links against =="
|
||||
sudo -n apt-get install -y -qq --no-install-recommends \
|
||||
build-essential cmake ninja-build pkg-config \
|
||||
libicu-dev libcurl4-openssl-dev libssl-dev libpng-dev libjpeg-dev ffmpeg
|
||||
|
||||
echo "== dropping Ubuntu's truncated rocWMMA headers =="
|
||||
if dpkg -s librocwmma-dev >/dev/null 2>&1; then
|
||||
sudo -n apt-get remove -y -qq librocwmma-dev
|
||||
fi
|
||||
|
||||
echo "== source: $GUFO_REPO @ $GUFO_TAG =="
|
||||
if [[ ! -d "$SRC_DIR/.git" ]]; then
|
||||
mkdir -p "$(dirname "$SRC_DIR")"
|
||||
git clone --depth 1 --branch "$GUFO_TAG" "$GUFO_REPO" "$SRC_DIR"
|
||||
fi
|
||||
git -C "$SRC_DIR" fetch --depth 1 origin "refs/tags/$GUFO_TAG"
|
||||
git -C "$SRC_DIR" checkout -q --detach "refs/tags/$GUFO_TAG"
|
||||
echo " $(git -C "$SRC_DIR" rev-parse HEAD)"
|
||||
check_pin
|
||||
|
||||
echo "== configure + build (release, HIP gfx1151) =="
|
||||
cd "$SRC_DIR"
|
||||
toolchain="--gcc-install-dir=$GCC_DIR"
|
||||
cmake --preset release \
|
||||
-DCMAKE_INSTALL_PREFIX="$PREFIX" \
|
||||
-DCMAKE_HIP_FLAGS="$toolchain" \
|
||||
-DCMAKE_EXE_LINKER_FLAGS="$toolchain"
|
||||
cmake --build --preset release --parallel "$JOBS"
|
||||
cmake --install build/release
|
||||
|
||||
echo "== verify =="
|
||||
"$PREFIX/bin/gufo" diagnose
|
||||
echo "frank-large runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)"
|
||||
+29
-15
@@ -1077,18 +1077,25 @@ deploy_anubis() {
|
||||
wait_container_healthy anubis 60 || return 1
|
||||
}
|
||||
|
||||
# frank-large is the stock llama.cpp prebuilt (unsloth b11030, ROCm
|
||||
# gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2
|
||||
# drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU,
|
||||
# as a ROOT unit since the 2026-09-19 cutover: the runtime is a prebuilt
|
||||
# binary, so there is no container image to install and no generated unit
|
||||
# to dodge. relink_unit points /etc/systemd/system/frank-large.service at
|
||||
# the repo file and boot persistence is that unit's [Install] section.
|
||||
# frank-large is gufo (gufo-org/gufo), the Strix Halo inference engine built
|
||||
# from the pinned v0.1.1 tag by scripts/build-gufo.sh, serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter
|
||||
# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU. It is a
|
||||
# ROOT unit: the runtime is a native binary linked against the host ROCm,
|
||||
# so there is no container image to install and no generated unit to dodge.
|
||||
# relink_unit points /etc/systemd/system/frank-large.service at the repo
|
||||
# file and boot persistence is that unit's [Install] section.
|
||||
#
|
||||
# It replaces the Kairic Edge runtime (ciru-ai/ROCmFPX), which had
|
||||
# replaced the unsloth llama.cpp prebuilt serving
|
||||
# unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS. The port gate is the same as
|
||||
# every cutover since 2026-09-16: nothing starts until 7770 is released.
|
||||
# The runtime gate matters as much as the payload gate: gufo is built from
|
||||
# source on this host (there is no prebuilt release), so a machine whose
|
||||
# /home/dank/GitPackages tree was never built has a unit that can never
|
||||
# become healthy, and a green deploy would leave the endpoint dark.
|
||||
#
|
||||
# It replaces the stock llama.cpp prebuilt (unsloth b11030), which had
|
||||
# replaced the Kairic Edge runtime (ciru-ai/ROCmFPX). The port gate is the
|
||||
# same as every cutover since 2026-09-16: nothing starts until 7770 is
|
||||
# released.
|
||||
LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}"
|
||||
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf}"
|
||||
LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
|
||||
@@ -1126,6 +1133,11 @@ deploy_llm_large() {
|
||||
# Refuse rather than restart-loop. A unit whose runtime or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the deploy log to say why.
|
||||
if [[ ! -x "$LLM_LARGE_RUNTIME" ]]; then
|
||||
echo "ERROR: gufo runtime absent at $LLM_LARGE_RUNTIME" >&2
|
||||
echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
local missing=0 f
|
||||
for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
@@ -1140,10 +1152,12 @@ deploy_llm_large() {
|
||||
# On a re-deploy the predecessor is THIS unit: stop it before waiting on
|
||||
# the port, or the wait deadlocks on the server it just stopped serving.
|
||||
# Then sweep a stray manual run of the SAME model -- that is how a bench
|
||||
# harness leaves the port busy. Match the model path, not the binary name:
|
||||
# frank-small on this host is also a llama-server and must not be killed.
|
||||
# The pattern never appears in the deploying shell's own command line, so
|
||||
# -f is safe here (it is not in `bash deploy-remote.sh`).
|
||||
# harness leaves the port busy, and the pattern matches the gufo command
|
||||
# line because it carries the model path. Match the model path, not a
|
||||
# binary name: frank-small on this host serves a different GGUF over
|
||||
# llama-server and must not be killed. The pattern never appears in the
|
||||
# deploying shell's own command line, so -f is safe here (it is not in
|
||||
# `bash deploy-remote.sh`).
|
||||
run_privileged systemctl stop frank-large.service 2>/dev/null || true
|
||||
pkill -f 'Qwen3.8-27B-UD-Q8_K_XL' 2>/dev/null || true
|
||||
# Nothing may start until the predecessor has released the port. On
|
||||
|
||||
+18
-18
@@ -1,6 +1,5 @@
|
||||
# frank-large — stock llama.cpp (mainline, the unsloth b11030 prebuilt)
|
||||
# serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU
|
||||
# (gfx1151).
|
||||
# frank-large — gufo (gufo-org/gufo) serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU (gfx1151).
|
||||
#
|
||||
# These are the tunables read by scripts/serve-frank-large.sh. The flag
|
||||
# list is owned by the serve script; this file only sets fleet values.
|
||||
@@ -10,27 +9,28 @@
|
||||
# exception because the serve script itself consumes it for --host.
|
||||
|
||||
# The payload (scripts/download-frank-large.sh): the serve script refuses to
|
||||
# start without the main GGUF and the DFlash2 drafter.
|
||||
# start without the main GGUF and the DFlash2 drafter. Both files are the
|
||||
# same ones the llama.cpp deployment used — the cutover changed the RUNTIME,
|
||||
# not the weights.
|
||||
MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
|
||||
|
||||
# The stock llama.cpp runtime (the unsloth b11030 prebuilt).
|
||||
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
|
||||
# The gufo runtime, built from source by scripts/build-gufo.sh.
|
||||
RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1
|
||||
|
||||
# Fleet endpoint: the same port and alias the prior deployments served, so
|
||||
# Fleet endpoint: the same port and alias every prior frank-large served, so
|
||||
# no client, probe or models.yml entry changes address.
|
||||
ALIAS=frank-large
|
||||
HOST=0.0.0.0
|
||||
PORT=7770
|
||||
|
||||
# One stream, full 262144 context (the model's native window).
|
||||
# One session, full 262144 context (the model's native window).
|
||||
CONTEXT_SIZE=262144
|
||||
PARALLEL_SLOTS=1
|
||||
SESSIONS=1
|
||||
|
||||
# DFlash2 spec decode (a block-diffusion drafter) at depth 4. The drafter
|
||||
# has a sliding window of 2048 (small draft KV, ~40 MiB).
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
|
||||
# Prompt-cache budget: 8 GiB. The model is a 27B hybrid (16 full attention
|
||||
# + 48 Gated DeltaNet linear attention); its DeltaNet state is smaller than
|
||||
# a primarily-DeltaNet model, so the fleet's 12.8 GiB DeltaNet floor does
|
||||
# not apply.
|
||||
CACHE_RAM=8192
|
||||
# DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft
|
||||
# block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is
|
||||
# 7, chosen over llama.cpp's fixed depth 4 because gufo's adaptive policy
|
||||
# shortens the block when acceptance drops) and DRAFT_POLICY is the block
|
||||
# length controller.
|
||||
DRAFT_TOKENS=7
|
||||
DRAFT_POLICY=adaptive
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=stock llama.cpp (unsloth b11030, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, fp8 KV, 262144 ctx / 1 slot, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
|
||||
Description=gufo (gufo-org/gufo, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
@@ -1,32 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on
|
||||
# Dankclaw's Strix Halo iGPU (gfx1151), using stock llama.cpp
|
||||
# (mainline, the unsloth b11030 prebuilt).
|
||||
# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on Dankclaw's
|
||||
# Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the inference
|
||||
# engine built specifically for this hardware.
|
||||
#
|
||||
# Runtime: /home/dank/GitPackages/unsloth-llama-b11030/llama-server
|
||||
# (mainline, ROCm gfx1151 prebuilt). The build supports the qwen3_5
|
||||
# architecture and the DFlash2 drafter (llama_model_qwen35,
|
||||
# llama_model_dflash, common_speculative_impl_draft_dflash).
|
||||
# Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source
|
||||
# against the host ROCm by scripts/build-gufo.sh. It replaces the stock
|
||||
# llama.cpp prebuilt (unsloth b11030) that served this endpoint from
|
||||
# 2026-09-21; payload, port and alias are unchanged, so no client moved.
|
||||
#
|
||||
# Shape: one stream, full 262144 context (the model's native window), all
|
||||
# layers on the iGPU (--n-gpu-layers 999). The 27B dense model (31.46 GiB GGUF) + the
|
||||
# DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) + compute buffers fits the
|
||||
# iGPU's GTT aperture (~114 GiB).
|
||||
# Shape: one session, full 262144 context (the model's native window). The
|
||||
# 27B dense weights (29.30 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
|
||||
# compute buffers fit the iGPU's GTT aperture (~114 GiB); gufo's own
|
||||
# Q8/AR table measures 50.58 GiB peak at this context capacity.
|
||||
#
|
||||
# Spec method: DFlash2 (a block-diffusion drafter) at depth 4. The drafter
|
||||
# has a sliding window of 2048 (small draft KV, ~40 MiB).
|
||||
# Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block
|
||||
# policy, capped at DRAFT_TOKENS proposals per step. The drafter is the
|
||||
# z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048
|
||||
# (small draft KV).
|
||||
#
|
||||
# KV: q8_0 quantized (--cache-type-k q8_0 --cache-type-v q8_0). The unsloth b11030
|
||||
# build does not support fp8 KV (allowed: f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/
|
||||
# q5_1); q8_0 is the highest-precision 8-bit option and preserves the ~8 GiB footprint.
|
||||
# Sampling: greedy (--temperature 0). That is what this endpoint has always
|
||||
# served, and it keeps the swap to a pure runtime change; gufo's per-model
|
||||
# recommendation for Qwen3.8 thinking is sampled (1.0 / 0.95 / top-k 20) and
|
||||
# is a separate decision.
|
||||
#
|
||||
# Text-only (no mmproj) — matching the prior frank-large.
|
||||
# Template: gufo compiles the official Qwen Jinja into the binary and takes
|
||||
# no --chat-template-file. Its Qwen27B default is thinking ON at xhigh effort
|
||||
# with prior reasoning preserved, which is the contract the fleet already
|
||||
# depends on, so --think auto + --preserve-thinking on is spelled out here
|
||||
# rather than left implicit.
|
||||
#
|
||||
# The Qwen3.8 family template keeps prior thinking blocks across turns when
|
||||
# preserve_thinking is set; pass --jinja and the kwarg so multi-turn agents
|
||||
# do not lose the trail. The template gates reasoning_effort to {xhigh,
|
||||
# medium, low} — the harness folds its levels onto that set in models.yml
|
||||
# (reasoningEffortMap).
|
||||
# Text-only: gufo auto-discovers a mmproj beside the model and this payload
|
||||
# has none, so the endpoint stays text-only.
|
||||
#
|
||||
# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
|
||||
# it; anything not set there falls through to the defaults below.
|
||||
@@ -41,22 +45,23 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) ------------------
|
||||
: "${RUNTIME_DIR:=/home/dank/GitPackages/unsloth-llama-b11030}"
|
||||
: "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
|
||||
: "${ALIAS:=frank-large}"
|
||||
: "${HOST:=0.0.0.0}"
|
||||
: "${PORT:=7770}"
|
||||
: "${CONTEXT_SIZE:=262144}"
|
||||
: "${PARALLEL_SLOTS:=1}"
|
||||
: "${SPEC_DRAFT_N_MAX:=4}"
|
||||
: "${CACHE_RAM:=8192}"
|
||||
TEMPLATE_FILE="${TEMPLATE_FILE:-$SCRIPT_DIR/qwen-sharp-v22.4.0.jinja}"
|
||||
: "${SESSIONS:=1}"
|
||||
: "${DRAFT_TOKENS:=7}"
|
||||
: "${DRAFT_POLICY:=adaptive}"
|
||||
|
||||
SERVER_BIN="$RUNTIME_DIR/llama-server"
|
||||
GUFO_BIN="$RUNTIME_DIR/bin/gufo"
|
||||
MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
||||
DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
|
||||
if [[ ! -x "$SERVER_BIN" ]]; then
|
||||
echo "runtime not found at $SERVER_BIN" >&2
|
||||
if [[ ! -x "$GUFO_BIN" ]]; then
|
||||
echo "runtime not found at $GUFO_BIN" >&2
|
||||
echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
|
||||
@@ -66,42 +71,28 @@ for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
|
||||
fi
|
||||
done
|
||||
|
||||
# --- env (the unsloth build's ROCm setup) --------------------------------
|
||||
# --- env -------------------------------------------------------------------
|
||||
# gufo links against the host ROCm through its RPATH (/opt/rocm ->
|
||||
# /etc/alternatives/rocm), so no LD_LIBRARY_PATH is needed. The device is
|
||||
# already gfx1151 — `gufo diagnose` reports it natively — so no
|
||||
# HSA_OVERRIDE_GFX_VERSION either; the override existed for the prebuilt
|
||||
# llama.cpp runtime and can only misreport the arch here.
|
||||
export HIP_VISIBLE_DEVICES=0
|
||||
export HSA_OVERRIDE_GFX_VERSION=11.5.1
|
||||
export LD_LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/llvm/lib:${LD_LIBRARY_PATH:-}"
|
||||
|
||||
# --- the serve command ---------------------------------------------------
|
||||
ARGS=(
|
||||
--host "$HOST"
|
||||
--port "$PORT"
|
||||
--model "$MODEL_FILE"
|
||||
--alias frank-large
|
||||
--ctx-size "$CONTEXT_SIZE"
|
||||
--parallel "$PARALLEL_SLOTS"
|
||||
--n-gpu-layers 999
|
||||
--jinja
|
||||
--metrics
|
||||
--chat-template-file "$TEMPLATE_FILE"
|
||||
--chat-template-kwargs '{"preserve_thinking":true}'
|
||||
--cache-type-k q8_0
|
||||
--cache-type-v q8_0
|
||||
--cache-ram "$CACHE_RAM"
|
||||
# DFlash2 spec decode (a block-diffusion drafter)
|
||||
--spec-type draft-dflash
|
||||
--model-draft "$DRAFTER_FILE"
|
||||
--spec-draft-n-max "$SPEC_DRAFT_N_MAX"
|
||||
--spec-draft-n-min 0
|
||||
--spec-draft-device ROCm0
|
||||
# Greedy sampling (the DFlash2 drafter needs greedy verification)
|
||||
--temp 0
|
||||
--top-p 1.0
|
||||
--top-k 0
|
||||
--min-p 0.0
|
||||
# Reasoning (thoughts land in reasoning_content)
|
||||
--reasoning auto
|
||||
--reasoning-format deepseek
|
||||
--reasoning-budget -1
|
||||
)
|
||||
|
||||
exec "$SERVER_BIN" "${ARGS[@]}"
|
||||
# Server options precede the subcommand, model options follow it.
|
||||
exec "$GUFO_BIN" serve \
|
||||
--host "$HOST" \
|
||||
--port "$PORT" \
|
||||
--sessions "$SESSIONS" \
|
||||
llm \
|
||||
--model "$MODEL_FILE" \
|
||||
--served-model-name "$ALIAS" \
|
||||
--context "$CONTEXT_SIZE" \
|
||||
--speculative dflash2 \
|
||||
--dflash-model "$DRAFTER_FILE" \
|
||||
--draft-tokens "$DRAFT_TOKENS" \
|
||||
--draft-policy "$DRAFT_POLICY" \
|
||||
--temperature 0 \
|
||||
--think auto \
|
||||
--preserve-thinking on
|
||||
|
||||
@@ -101,8 +101,8 @@ cable between claw and server is required for 10G.
|
||||
|
||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||
- The GPU budget is the GTT aperture: ~114 GiB (`gtt_total` in `/sys/class/drm/card1/device/`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
|
||||
- The active service is `frank-large` on port 7770: stock llama.cpp (the unsloth b11030 prebuilt) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, q8_0 KV, DFlash2 drafter). See [[frank-large-unsloth-qwen38-27b-q8xl]].
|
||||
- The active runtime (the unsloth b11030 prebuilt) **bundles its own ROCm runtime** next to the binary (`RPATH=$ORIGIN/../lib`), so it has no host SDK gate — the system ROCm 7.2.4 is not on its path. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is no longer in the serving path.
|
||||
- The active service is `frank-large` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-large-gufo-qwen38-27b]].
|
||||
- The active runtime **links against the host ROCm** (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH, so it needs the host HIP stack rather than a bundled one — the opposite of the retired llama.cpp prebuilt, which carried its own. Building it needs the ROCm dev packages that are installed, and two workarounds recorded in the runbook (drop `librocwmma-dev`; pin `--gcc-install-dir` to GCC 15). The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is not in the serving path.
|
||||
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
|
||||
- 1.1 TB free disk space is ample for model storage.
|
||||
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
---
|
||||
type: reference
|
||||
title: "gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the one-session / full-window shape, what changed in the client contract versus llama.cpp, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- gufo
|
||||
- unsloth
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- strix-halo
|
||||
- gfx1151
|
||||
- q8
|
||||
- dflash2
|
||||
- rocm
|
||||
timestamp: 2026-09-28
|
||||
---
|
||||
|
||||
# frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2)
|
||||
|
||||
> **ACTIVE since 2026-09-28.** Replaces
|
||||
> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
|
||||
> b11030 prebuilt), which had replaced
|
||||
> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
|
||||
> alias (`frank-large`), **same weights**.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
||||
- **Runtime:** [gufo](https://github.com/gufo-org/gufo) — "the Strix Halo
|
||||
inference engine", a vertical engine built for exactly this hardware
|
||||
(Ryzen AI MAX+ 395 / Radeon 8060S, `gfx1151`, unified memory). Pinned at
|
||||
**v0.1.1** (commit `b0f8673`), built from source on the host to
|
||||
`/home/dank/GitPackages/gufo-v0.1.1/`.
|
||||
- **Service shape:** root systemd unit `scripts/frank-large.service`
|
||||
(`User=dank`, `LimitMEMLOCK=infinity`), launched by
|
||||
`scripts/serve-frank-large.sh` with tunables from
|
||||
`scripts/frank-large.env`.
|
||||
|
||||
gufo ships **no prebuilt Linux binaries** (the v0.1.0/v0.1.1 release assets
|
||||
are empty) and no Nix on this host, so the runtime is a source build. It is
|
||||
also young: v0.1.0 landed 2026-09-28, the same day as this cutover.
|
||||
|
||||
## Why this engine
|
||||
|
||||
gufo's own measurement of the two engines on the same Qwen3.8-27B targets
|
||||
(Strix Halo, `628ed18e` vs llama.cpp `b11069`, greedy, DFlash2):
|
||||
|
||||
| Q8_K_XL, DFlash2 | gufo | llama.cpp | delta |
|
||||
|---|---:|---:|---:|
|
||||
| prefill, depth 0 | 457.05 t/s | 333.85 t/s | **+36.9 %** |
|
||||
| prefill, depth 131,072 | 237.24 t/s | 131.80 t/s | **+80.0 %** |
|
||||
| decode, depth 0 (mixed) | 16.08 t/s | 15.00 t/s | +7.2 % |
|
||||
| decode, depth 32,768 (mixed) | 14.41 t/s | 14.44 t/s | −0.2 % |
|
||||
| decode, depth 65,536 (mixed) | 10.87 t/s | 12.62 t/s | **−13.9 %** |
|
||||
| decode, depth 131,072 (mixed) | 8.87 t/s | 11.62 t/s | **−23.7 %** |
|
||||
| cold load to HTTP-ready | 8.17 s | 36.47 s | **4.5× faster** |
|
||||
| peak memory @ 262,144 ctx | 50.58 GiB | 48.60 GiB | +4 % |
|
||||
|
||||
Read that honestly: **prefill, load time and short-context decode improve;
|
||||
deep-context decode on the 8-bit target gets worse.** The crossover on Q8 is
|
||||
around 32k of cached depth. frank-large runs one 262,144-token conversation,
|
||||
so a long agent session spends its life on the losing side of that line
|
||||
while a short turn wins. The Q4_K_XL target does not show the regression
|
||||
(gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to
|
||||
revisit the quantization separately from the runtime.
|
||||
|
||||
## Build
|
||||
|
||||
`scripts/build-gufo.sh` reproduces the runtime end to end. It clones the
|
||||
pinned tag, configures the `release` preset (HIP, `--offload-arch=gfx1151`),
|
||||
builds, installs, and finishes with `gufo diagnose`. Two host facts it
|
||||
encodes, both found the hard way on 2026-09-28:
|
||||
|
||||
- **Ubuntu's `librocwmma-dev` must not be installed.** It ships a truncated
|
||||
rocWMMA header set (the public headers without `rocwmma/internal/`) into
|
||||
`/usr/include`, where it shadows AMD's complete copy under
|
||||
`/opt/rocm/include`. The first build died on `'internal/accessors.hpp'
|
||||
file not found` from `/usr/include/rocwmma/rocwmma.hpp`, and **a `-I`
|
||||
cannot fix it**: in HIP mode the host compilation pass searches the system
|
||||
directories first. The package has no reverse dependencies; the script
|
||||
removes it.
|
||||
- **`--gcc-install-dir` must point at GCC 15.** Ubuntu 26.04 has
|
||||
`/usr/include/c++/16`, and the ROCm clang picks the newest tree, whose
|
||||
`<atomic_wait.h>` references `std::__detail::__notify_impl` at a symbol
|
||||
version the installed `libstdc++.so.6` (GLIBCXX_3.4.35) does not export.
|
||||
The link then fails with `undefined symbol`. Pinning the headers to the
|
||||
GCC the runtime library came from fixes it.
|
||||
|
||||
Verify an installed runtime with `bash scripts/build-gufo.sh --verify`, or
|
||||
`gufo diagnose` directly — it reports `[PASS]` plus the detected
|
||||
architecture, driver and queue budget.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — 27B dense,
|
||||
29.30 GiB. **Unchanged by the cutover**, still fetched and sha256-verified
|
||||
by `scripts/download-frank-large.sh`.
|
||||
- **Context:** one session, full 262,144 tokens (the model's native window),
|
||||
`SESSIONS=1`. The fleet's single stream is unchanged.
|
||||
- **Spec method:** DFlash2, gufo's **adaptive** block policy
|
||||
(`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the same
|
||||
z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB).
|
||||
- **Sampling:** greedy (`--temperature 0`), as every prior frank-large
|
||||
served. gufo's per-model recommendation for Qwen3.8 thinking is *sampled*
|
||||
(1.0 / 0.95 / top-k 20) — that is a separate decision, deliberately not
|
||||
bundled into a runtime swap.
|
||||
- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
|
||||
with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
|
||||
|
||||
## Client contract — what actually changed
|
||||
|
||||
Everything the fleet depended on still holds: port 7770, alias `frank-large`,
|
||||
`reasoning_content`, `tool_calls`, streaming SSE, and text-only (gufo
|
||||
auto-discovers a `mmproj` beside the model; this payload has none).
|
||||
|
||||
The differences worth knowing:
|
||||
|
||||
- **No custom chat template.** gufo compiles the official Qwen Jinja into the
|
||||
binary and takes no `--chat-template-file`. The Qwen-Sharp v22.4.0 file
|
||||
this endpoint used with llama.cpp is **no longer in the command line** —
|
||||
the 2026-09-21 "73,600-token prompt" incident cannot recur, because there
|
||||
is no fallback template to land in. Thinking defaults to on at `xhigh`
|
||||
with prior reasoning preserved (`--think auto --preserve-thinking on`,
|
||||
spelled out in the serve script).
|
||||
- **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`,
|
||||
`low`, `medium`, `high`, `xhigh`, `max`). Qwen-Sharp rejected anything
|
||||
outside `{xhigh, medium, low}` with an HTTP 500, which is why `models.yml`
|
||||
still carries a `reasoningEffortMap` fold for this model. **The fold is
|
||||
kept on purpose**: removing it would silently move every `high` role from
|
||||
`xhigh` to `high` effort in the same commit as the runtime swap. Drop it as
|
||||
its own change once the effort curve is a decision rather than a side
|
||||
effect.
|
||||
- **No KV-cache quantization flag.** gufo picks its own KV dtype; there is
|
||||
no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it.
|
||||
Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in
|
||||
and currently off.
|
||||
- **Health/readiness split:** `GET /health` is process liveness and is what
|
||||
the deployer gates on; `GET /ready` is model/backend readiness.
|
||||
`GET /metrics` is Prometheus-format.
|
||||
- **Long bodies:** `--max-request-bytes` defaults to 8 MiB and unknown
|
||||
request fields are rejected rather than ignored.
|
||||
|
||||
## Failure modes worth knowing
|
||||
|
||||
- **A missing runtime is a deploy error, not a restart loop.** The handler
|
||||
gates on `LLM_LARGE_RUNTIME` *before* it stops anything, and the serve
|
||||
script exits with a `build-gufo.sh` pointer rather than exec'ing.
|
||||
- **Host memory is the ceiling, and it is shared.** frank-large and ComfyUI
|
||||
both hold gfx1151 compute queues. gufo reads
|
||||
`/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself
|
||||
(`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device
|
||||
time-slices instead of failing; four HIP processes is the device ceiling.
|
||||
- **Load is seconds, not minutes** (8 s measured cold), so a redeploy no
|
||||
longer spends 20 minutes in `LLM_WAIT`.
|
||||
- **A v0.1.x runtime is a moving target.** Rebuilds are not automatic —
|
||||
`RUNTIME_DIR` names a version, and the deployer will not silently
|
||||
substitute another.
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-large-unsloth-qwen38-27b-q8xl]] — the unsloth b11030 prebuilt is
|
||||
still at `/home/dank/GitPackages/unsloth-llama-b11030/` and the payload is
|
||||
the same directory the current runtime reads. Roll back by reverting
|
||||
`RUNTIME_DIR` (and the gufo-only tunables) in `scripts/frank-large.env`,
|
||||
restoring the serve script's llama.cpp command, and redeploying.
|
||||
|
||||
# Citations
|
||||
|
||||
- Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT)
|
||||
- Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo
|
||||
- API contract: `docs/SERVER.md`; flags: `docs/CLI.md`
|
||||
- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`;
|
||||
tunables: `scripts/frank-large.env`
|
||||
- Payload: `scripts/download-frank-large.sh`
|
||||
- Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]]
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
|
||||
description: "The RETIRED frank-large runbook (2026-09-21 → 2026-09-28; replaced by [[frank-large-gufo-qwen38-27b]], and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- llama.cpp
|
||||
@@ -19,10 +19,12 @@ timestamp: 2026-09-21
|
||||
|
||||
# frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
|
||||
|
||||
> **ACTIVE since 2026-09-21.** Replaces [[frank-large-kairic-edge-qwen38-27b]]
|
||||
> (Kairic Edge IU4 W4A4, now the rollback target) and
|
||||
> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it preceded).
|
||||
> Same port (7770), same alias (`frank-large`), same client contract surface.
|
||||
> **RETIRED 2026-09-28**, replaced by [[frank-large-gufo-qwen38-27b]] (the
|
||||
> gufo engine, same weights, same port and alias); the prebuilt is intact on
|
||||
> disk as the rollback target. ACTIVE from 2026-09-21, when it replaced
|
||||
> [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge IU4 W4A4) and
|
||||
> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it
|
||||
> preceded).
|
||||
|
||||
## Identity
|
||||
|
||||
|
||||
@@ -139,10 +139,18 @@ frank-small, port 7771]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — ACTIVE frank-large runbook
|
||||
since 2026-09-21 (root systemd unit, port 7770): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
- [[frank-large-gufo-qwen38-27b]] — ACTIVE frank-large runbook since
|
||||
2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the
|
||||
Strix Halo inference engine, built from source on Dankclaw, serving
|
||||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2
|
||||
Q4_K_M drafter (adaptive block policy) on the gfx1151 iGPU — one full
|
||||
262,144-token conversation, greedy, text-only, the two from-source build
|
||||
workarounds, the client-contract delta, the measured prefill/decode trade
|
||||
against llama.cpp, troubleshooting
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
|
||||
Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
|
||||
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
|
||||
v22.4.0 template, the 114 GiB GTT memory model, client contract,
|
||||
|
||||
@@ -190,13 +190,21 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
|
||||
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
|
||||
rollback record)
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — the ACTIVE frank-large runbook
|
||||
since 2026-09-21 (root systemd unit, port 7770): stock llama.cpp
|
||||
- [[frank-large-gufo-qwen38-27b]] — the ACTIVE frank-large runbook since
|
||||
2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1,
|
||||
the Strix Halo inference engine, built from source on Dankclaw and
|
||||
serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the
|
||||
z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — one 262,144-token
|
||||
conversation, adaptive block policy, greedy, text-only, the two
|
||||
from-source build workarounds, what changed in the client contract, the
|
||||
measured prefill/decode trade against llama.cpp
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the z-lab
|
||||
DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one 262,144-token
|
||||
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp v22.4.0
|
||||
template, the 114 GiB GTT memory model, client contract
|
||||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter
|
||||
(depth 4) on the gfx1151 iGPU — one 262,144-token conversation, q8_0 KV
|
||||
(the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the
|
||||
114 GiB GTT memory model, client contract
|
||||
- [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (replaced by
|
||||
[[frank-large-unsloth-qwen38-27b-q8xl]]; same-day swap): the Kairic Edge
|
||||
runtime (ciru-ai/ROCmFPX kairic-edge-qwen38-27b-v1.2, TheRock 7.15.0,
|
||||
|
||||
+213
-86
@@ -1,31 +1,84 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-large root unit on dankclaw.
|
||||
#
|
||||
# The 2026-09-21 cutover moved this service from the Kairic Edge runtime
|
||||
# (ciru-ai/ROCmFPX) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge onto the
|
||||
# stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter
|
||||
# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M), still as a ROOT unit: a
|
||||
# prebuilt runtime has no container image to install and no generated
|
||||
# unit to dodge. This seam pins what a flag list cannot show: the
|
||||
# operator's one-stream / full-window shape, the payload the serve script
|
||||
# refuses to start without, the DFlash2 spec wiring, fp8 KV, greedy
|
||||
# sampling, a deploy handler that gates on the port, and the fact that
|
||||
# the sweep for a stray benchmark run cannot take frank-small down with
|
||||
# it.
|
||||
|
||||
# Since 2026-09-28 this endpoint is served by gufo (gufo-org/gufo), the
|
||||
# Strix Halo inference engine, built from the pinned v0.1.1 tag and serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter. It replaced
|
||||
# the stock llama.cpp prebuilt (unsloth b11030) on the same port, the same
|
||||
# alias and the same weights.
|
||||
#
|
||||
# Most of this seam runs the serve script against a stub engine and asserts
|
||||
# on the argv it actually execs, so what is pinned is the command the
|
||||
# operator gets rather than the text of the script: a tunable renamed,
|
||||
# dropped, or silently defaulted away fails here the way it would fail a
|
||||
# deployment.
|
||||
load helpers
|
||||
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
|
||||
UNIT="$REPO_ROOT/scripts/frank-large.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
|
||||
BUILD="$REPO_ROOT/scripts/build-gufo.sh"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "the frank-large unit, env and serve files exist where the deployer installs them from" {
|
||||
setup() {
|
||||
STAGE="$(mktemp -d)"
|
||||
RUNTIME_DIR="$STAGE/gufo"
|
||||
MODEL_DIR="$STAGE/models"
|
||||
mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR"
|
||||
# A stub engine: the serve script execs it, so $@ is exactly what gufo
|
||||
# would be invoked with.
|
||||
cat > "$RUNTIME_DIR/bin/gufo" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$@"
|
||||
STUB
|
||||
chmod +x "$RUNTIME_DIR/bin/gufo"
|
||||
: > "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
||||
: > "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
STUB_ENV="$STAGE/stub.env"
|
||||
cat > "$STUB_ENV" <<EOF
|
||||
RUNTIME_DIR=$RUNTIME_DIR
|
||||
MODEL_DIR=$MODEL_DIR
|
||||
EOF
|
||||
}
|
||||
|
||||
teardown() {
|
||||
if [ -n "${STAGE:-}" ]; then
|
||||
rm -rf "$STAGE"
|
||||
fi
|
||||
}
|
||||
|
||||
# serve_argv [env-file] — run the serve script and capture the engine's argv,
|
||||
# one token per line, in $output. No argument exercises the built-in defaults
|
||||
# with the stub's paths.
|
||||
serve_argv() {
|
||||
local envfile="${1:-$STUB_ENV}"
|
||||
if [ -n "$envfile" ]; then
|
||||
FRANK_LARGE_ENV_FILE="$envfile" run bash "$SERVE"
|
||||
else
|
||||
run env FRANK_LARGE_ENV_FILE="$STAGE/absent.env" bash "$SERVE"
|
||||
fi
|
||||
}
|
||||
|
||||
# argv_has <token> — exact token match, so --port never satisfies a test
|
||||
# looking for a different flag.
|
||||
argv_has() {
|
||||
grep -qxF -- "$1" <<<"$output"
|
||||
}
|
||||
|
||||
# argv_value <flag> — the token following <flag>, empty if the flag is absent.
|
||||
argv_value() {
|
||||
local i
|
||||
i="$(grep -nx -- "$1" <<<"$output" | cut -d: -f1)"
|
||||
[ -n "$i" ] || return 0
|
||||
sed -n "$((i + 1))p" <<<"$output"
|
||||
}
|
||||
|
||||
@test "the frank-large unit, env, serve and build files exist where the deployer installs them from" {
|
||||
[ -f "$ENV_FILE" ]
|
||||
[ -f "$UNIT" ]
|
||||
[ -x "$SERVE" ]
|
||||
[ -x "$BUILD" ]
|
||||
}
|
||||
|
||||
@test "frank-large is boot-persistent, self-healing and unprivileged" {
|
||||
@@ -33,65 +86,108 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
# The engine pins staging buffers; the memlock budget must be unbounded
|
||||
# or the load fails on the RLIMIT.
|
||||
# The engine pins buffers; the memlock budget must be unbounded or the
|
||||
# load fails on the RLIMIT.
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
# The unit runs the repo's own serve script with the repo's env.
|
||||
grep -q 'ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh' "$UNIT"
|
||||
grep -q 'FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the operator's shape is one stream of the full 262144 window with DFlash2 depth 4" {
|
||||
local ctx slots
|
||||
ctx="$(sed -nE 's/^CONTEXT_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
slots="$(sed -nE 's/^PARALLEL_SLOTS=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
@test "the fleet endpoint and alias survive the cutover" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
# frank-large at :7770, whatever engine answers it.
|
||||
[ "$(argv_value --served-model-name)" = "frank-large" ]
|
||||
[ "$(argv_value --port)" = "7770" ]
|
||||
[ "$(argv_value --host)" = "0.0.0.0" ]
|
||||
}
|
||||
|
||||
@test "the built-in defaults are one session of the full 262144 window with DFlash2" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
# 262144 is the model's native window (the operator's call, 2026-09-21).
|
||||
[ "$ctx" -eq 262144 ]
|
||||
# One stream.
|
||||
[ "$slots" -eq 1 ]
|
||||
# DFlash2 depth 4 (the drafter's block-diffusion depth).
|
||||
grep -q '^SPEC_DRAFT_N_MAX=4$' "$ENV_FILE"
|
||||
[ "$(argv_value --context)" -eq 262144 ]
|
||||
# One session is the single stream every prior frank-large served.
|
||||
[ "$(argv_value --sessions)" -eq 1 ]
|
||||
[ "$(argv_value --speculative)" = "dflash2" ]
|
||||
# A draft ceiling of 0 would silently serve the autoregressive rate.
|
||||
[ "$(argv_value --draft-tokens)" -gt 0 ]
|
||||
}
|
||||
|
||||
@test "the shipped env file, not just the defaults, serves the deployed shape" {
|
||||
# The defaults are a floor; systemd sources the env file. Only the two
|
||||
# on-host paths are redirected at the stub, so every fleet value in the
|
||||
# shipped file is the one under test.
|
||||
sed -e "s#^RUNTIME_DIR=.*#RUNTIME_DIR=$RUNTIME_DIR#" \
|
||||
-e "s#^MODEL_DIR=.*#MODEL_DIR=$MODEL_DIR#" \
|
||||
"$ENV_FILE" > "$STAGE/shipped.env"
|
||||
serve_argv "$STAGE/shipped.env"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --served-model-name)" = "frank-large" ]
|
||||
[ "$(argv_value --host)" = "0.0.0.0" ]
|
||||
[ "$(argv_value --port)" = "7770" ]
|
||||
[ "$(argv_value --context)" -eq 262144 ]
|
||||
[ "$(argv_value --sessions)" -eq 1 ]
|
||||
[ "$(argv_value --speculative)" = "dflash2" ]
|
||||
[ "$(argv_value --draft-tokens)" -gt 0 ]
|
||||
[ "$(argv_value --draft-policy)" = "adaptive" ]
|
||||
}
|
||||
|
||||
@test "the engine is pointed at the payload that is actually on disk" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ -f "$(argv_value --model)" ]
|
||||
# A drafter path that does not exist would load the target and decode at
|
||||
# the autoregressive rate, which reads as "the new runtime is slow".
|
||||
[ -f "$(argv_value --dflash-model)" ]
|
||||
[ "$(argv_value --model)" != "$(argv_value --dflash-model)" ]
|
||||
}
|
||||
|
||||
@test "frank-large stays text-only" {
|
||||
# The new model is text-only (no vision projector). The fleet's
|
||||
# image path is frank-small; models.yml declares frank-large text-only.
|
||||
! grep -q -- '--mmproj' "$SERVE"
|
||||
# The payload has no mmproj, so an --mmproj would point at nothing. The
|
||||
# fleet's image path is frank-small; models.yml declares frank-large
|
||||
# text-only.
|
||||
serve_argv
|
||||
! argv_has --mmproj
|
||||
grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
}
|
||||
|
||||
@test "the serve script pins the stock llama.cpp runtime and refuses a missing binary" {
|
||||
grep -q 'unsloth-llama-b11030' "$SERVE"
|
||||
grep -q 'llama-server' "$SERVE"
|
||||
# A missing runtime or payload is a deploy error with a pointer, not a
|
||||
# restart loop.
|
||||
grep -q 'runtime not found' "$SERVE"
|
||||
grep -q 'missing asset' "$SERVE"
|
||||
# The Qwen3.8 template keeps prior thinking blocks across turns only
|
||||
# when the kwarg is set.
|
||||
grep -q 'preserve_thinking' "$SERVE"
|
||||
# The chat template file is wired (not just the kwarg) — the 73k-token
|
||||
# fallback incident (2026-09-21) was a missing --chat-template-file.
|
||||
grep -q 'chat-template-file' "$SERVE"
|
||||
@test "the engine is asked for greedy decoding and for preserved prior reasoning" {
|
||||
# Greedy is what this endpoint has always served; a silent switch to
|
||||
# sampled output would be a model-behaviour change disguised as a
|
||||
# runtime swap. Prior thinking blocks must survive across turns or a
|
||||
# multi-turn agent loses its own reasoning between tool calls.
|
||||
serve_argv
|
||||
[ "$(argv_value --temperature)" = "0" ]
|
||||
[ "$(argv_value --preserve-thinking)" = "on" ]
|
||||
}
|
||||
|
||||
@test "the serve script wires the DFlash2 spec method and the drafter" {
|
||||
# The new model uses DFlash2 (block-diffusion drafter) at depth 4.
|
||||
grep -q -- '--spec-type draft-dflash' "$SERVE"
|
||||
grep -q -- '--model-draft' "$SERVE"
|
||||
grep -q -- '--spec-draft-n-max' "$SERVE"
|
||||
# The drafter runs on the iGPU (ROCm0).
|
||||
grep -q -- '--spec-draft-device ROCm0' "$SERVE"
|
||||
@test "the serve script refuses a missing runtime with the build pointer" {
|
||||
rm -f "$RUNTIME_DIR/bin/gufo"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"runtime not found"* ]]
|
||||
[[ "$output" == *"build-gufo.sh"* ]]
|
||||
# A restart loop would be the alternative; nothing may be exec'd.
|
||||
! argv_has serve
|
||||
}
|
||||
|
||||
@test "the serve script uses q8_0 KV and greedy sampling" {
|
||||
# q8_0 KV — the unsloth b11030 build does not support fp8 KV (allowed:
|
||||
# f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the
|
||||
# highest-precision 8-bit option and preserves the ~8 GiB footprint.
|
||||
grep -q -- '--cache-type-k q8_0' "$SERVE"
|
||||
grep -q -- '--cache-type-v q8_0' "$SERVE"
|
||||
# Greedy sampling (DFlash2 needs greedy verification).
|
||||
grep -q -- '--temp 0' "$SERVE"
|
||||
@test "the serve script refuses a missing drafter with the fetch pointer" {
|
||||
rm -f "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"missing asset"* ]]
|
||||
[[ "$output" == *"download-frank-large.sh"* ]]
|
||||
! argv_has serve
|
||||
}
|
||||
|
||||
@test "the serve script refuses a missing target and names the file" {
|
||||
rm -f "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"missing asset"* ]]
|
||||
[[ "$output" == *"Qwen3.8-27B-UD-Q8_K_XL.gguf"* ]]
|
||||
}
|
||||
|
||||
@test "the payload is fetched resumably and verified against declared sizes + sha256" {
|
||||
@@ -103,41 +199,72 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
grep -q 'SHA256 MISMATCH' "$FETCH"
|
||||
}
|
||||
|
||||
@test "the fleet endpoint and alias survive the cutover" {
|
||||
grep -q '^HOST=0.0.0.0$' "$ENV_FILE"
|
||||
grep -q '^PORT=7770$' "$ENV_FILE"
|
||||
grep -q '^MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL$' "$ENV_FILE"
|
||||
grep -q -- '--alias frank-large' "$SERVE"
|
||||
@test "the build script refuses a source tree that is not the pinned commit" {
|
||||
# gufo ships no prebuilt Linux binaries, so the runtime is built here.
|
||||
# A tag that moves under us would silently install different kernels
|
||||
# under the same RUNTIME_DIR, so the pin is enforced, not documented.
|
||||
local src="$STAGE/not-the-pinned-commit"
|
||||
mkdir -p "$src"
|
||||
git -C "$src" init -q
|
||||
git -C "$src" -c user.email=t@t -c user.name=t commit -q --allow-empty -m drift
|
||||
run env GUFO_SRC_DIR="$src" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"GUFO_COMMIT mismatch"* ]]
|
||||
}
|
||||
|
||||
@test "the build script reports a runtime that was never built" {
|
||||
# --verify on a host with no install must say so rather than exit 0 on
|
||||
# an empty check.
|
||||
run env GUFO_SRC_DIR="$STAGE/no-source-tree" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"no gufo at"* ]]
|
||||
}
|
||||
|
||||
@test "the deploy handler gates on the runtime, the payload and the port" {
|
||||
grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_RUNTIME' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_MODEL' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_DRAFT' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler will not stop a serving endpoint it cannot replace" {
|
||||
# The runtime gate must be live (a real conditional on an executable
|
||||
# path, not a branch that never fires) and it must return BEFORE the
|
||||
# stop, or a host whose gufo was never built takes frank-large down on
|
||||
# its way to reporting the failure.
|
||||
local large gate bail stop
|
||||
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
|
||||
[[ "$large" == *'build-gufo.sh'* ]]
|
||||
gate="$(grep -n 'if \[\[ ! -x "\$LLM_LARGE_RUNTIME" \]\]' <<<"$large" | head -1 | cut -d: -f1)"
|
||||
stop="$(grep -n 'systemctl stop frank-large' <<<"$large" | head -1 | cut -d: -f1)"
|
||||
[ -n "$gate" ]
|
||||
[ -n "$stop" ]
|
||||
bail="$(awk -v g="$gate" 'NR > g && /return 1/ { print NR; exit }' <<<"$large")"
|
||||
[ -n "$bail" ]
|
||||
[ "$bail" -lt "$stop" ]
|
||||
}
|
||||
|
||||
@test "the stray-run sweep cannot take frank-small down" {
|
||||
# frank-small on this host serves a different GGUF. The sweep must match
|
||||
# the model path, not a binary name, or a frank-large deploy restarts the
|
||||
# 3090 endpoint.
|
||||
grep -q "pkill -f 'Qwen3.8-27B-UD-Q8_K_XL'" "$REMOTE_DEPLOY"
|
||||
local large
|
||||
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
|
||||
! grep -q 'pkill -x llama-server' <<<"$large"
|
||||
! grep -q 'pkill -x gufo' <<<"$large"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the halogen quadlet without deleting it" {
|
||||
grep -q 'retire_halogen_large_quadlet' "$REMOTE_DEPLOY"
|
||||
grep -q 'HOME/.AGENTTRASH' "$REMOTE_DEPLOY"
|
||||
! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?.*frank-large' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler installs the root unit, gates the port and refuses a missing payload" {
|
||||
grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_MODEL' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_DRAFT' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the stray-run sweep cannot take frank-small down" {
|
||||
# frank-small on this host is ALSO a llama-server. The sweep must match
|
||||
# the model path, not the binary name, or a frank-large deploy restarts
|
||||
# the 3090 endpoint.
|
||||
grep -q "pkill -f 'Qwen3.8-27B-UD-Q8_K_XL'" "$REMOTE_DEPLOY"
|
||||
# ...and only inside the large handler: extract it and check there is no
|
||||
# bare binary-name pkill in it.
|
||||
local large
|
||||
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
|
||||
! grep -q 'pkill -x llama-server' <<<"$large"
|
||||
}
|
||||
|
||||
@test "the halogen quadlet is archived, not live" {
|
||||
# Nothing live ships for this endpoint as a container any more; the
|
||||
# archived copy is host state in an ignored .AGENTTRASH, not a tracked
|
||||
# file.
|
||||
[ ! -e "$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container" ]
|
||||
[ -f "$REPO_ROOT/dankclaw/.AGENTTRASH/frank-large.container.halogen" ]
|
||||
! grep -rq 'Image=.*halogen' "$REPO_ROOT/dankclaw"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user