Frank
aaef419a6d
gufo is a vertical inference engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large has swapped runtimes seven times in search of a better one. This one ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on the host and the payload, port, alias and client contract stay exactly where they are. Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1, built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4 through its RPATH — the opposite of the unsloth prebuilt it replaces, which carried its own — so two host workarounds are encoded in that script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set that shadows AMD's and cannot be overridden with -I in HIP mode, and the ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose libstdc++ is actually installed. Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL target and the same DFlash2 drafter, greedy, text-only. Speculation moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy capped at 7 proposals. Client-contract deltas, all documented: the chat template is compiled into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token failure mode is gone), reasoning_effort now takes the full OpenAI set (so models.yml's reasoningEffortMap fold is no longer required — kept anyway, so this commit does not also move every high role from xhigh to high effort), and there is no KV-quant or prompt-cache-RAM flag. The test seam now runs the serve script against a stub engine and asserts on the argv it actually execs, instead of grepping the script's text. The deploy handler gates on the runtime binary and returns before it stops the serving unit. Measured trade, from gufo's own Q8_K_XL tables against llama.cpp b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the 8-bit target the crossover is ~32k, and frank-large runs one 262,144 conversation. The Q4_K_XL target shows no such regression.
99 lines
3.8 KiB
Bash
Executable File
99 lines
3.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on Dankclaw's
|
|
# Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the inference
|
|
# engine built specifically for this hardware.
|
|
#
|
|
# Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source
|
|
# against the host ROCm by scripts/build-gufo.sh. It replaces the stock
|
|
# llama.cpp prebuilt (unsloth b11030) that served this endpoint from
|
|
# 2026-09-21; payload, port and alias are unchanged, so no client moved.
|
|
#
|
|
# Shape: one session, full 262144 context (the model's native window). The
|
|
# 27B dense weights (29.30 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
|
|
# compute buffers fit the iGPU's GTT aperture (~114 GiB); gufo's own
|
|
# Q8/AR table measures 50.58 GiB peak at this context capacity.
|
|
#
|
|
# Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block
|
|
# policy, capped at DRAFT_TOKENS proposals per step. The drafter is the
|
|
# z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048
|
|
# (small draft KV).
|
|
#
|
|
# Sampling: greedy (--temperature 0). That is what this endpoint has always
|
|
# served, and it keeps the swap to a pure runtime change; gufo's per-model
|
|
# recommendation for Qwen3.8 thinking is sampled (1.0 / 0.95 / top-k 20) and
|
|
# is a separate decision.
|
|
#
|
|
# Template: gufo compiles the official Qwen Jinja into the binary and takes
|
|
# no --chat-template-file. Its Qwen27B default is thinking ON at xhigh effort
|
|
# with prior reasoning preserved, which is the contract the fleet already
|
|
# depends on, so --think auto + --preserve-thinking on is spelled out here
|
|
# rather than left implicit.
|
|
#
|
|
# Text-only: gufo auto-discovers a mmproj beside the model and this payload
|
|
# has none, so the endpoint stays text-only.
|
|
#
|
|
# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
|
|
# it; anything not set there falls through to the defaults below.
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
|
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
|
if [[ -f "$ENV_FILE" ]]; then
|
|
set -a
|
|
source "$ENV_FILE"
|
|
set +a
|
|
fi
|
|
|
|
# --- defaults (used when the env file does not set them) ------------------
|
|
: "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}"
|
|
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
|
|
: "${ALIAS:=frank-large}"
|
|
: "${HOST:=0.0.0.0}"
|
|
: "${PORT:=7770}"
|
|
: "${CONTEXT_SIZE:=262144}"
|
|
: "${SESSIONS:=1}"
|
|
: "${DRAFT_TOKENS:=7}"
|
|
: "${DRAFT_POLICY:=adaptive}"
|
|
|
|
GUFO_BIN="$RUNTIME_DIR/bin/gufo"
|
|
MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
|
DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
|
|
|
if [[ ! -x "$GUFO_BIN" ]]; then
|
|
echo "runtime not found at $GUFO_BIN" >&2
|
|
echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
|
|
exit 1
|
|
fi
|
|
for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
|
|
if [[ ! -f "$f" ]]; then
|
|
echo "missing asset $f (run scripts/download-frank-large.sh)" >&2
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
# --- env -------------------------------------------------------------------
|
|
# gufo links against the host ROCm through its RPATH (/opt/rocm ->
|
|
# /etc/alternatives/rocm), so no LD_LIBRARY_PATH is needed. The device is
|
|
# already gfx1151 — `gufo diagnose` reports it natively — so no
|
|
# HSA_OVERRIDE_GFX_VERSION either; the override existed for the prebuilt
|
|
# llama.cpp runtime and can only misreport the arch here.
|
|
export HIP_VISIBLE_DEVICES=0
|
|
|
|
# --- the serve command ---------------------------------------------------
|
|
# Server options precede the subcommand, model options follow it.
|
|
exec "$GUFO_BIN" serve \
|
|
--host "$HOST" \
|
|
--port "$PORT" \
|
|
--sessions "$SESSIONS" \
|
|
llm \
|
|
--model "$MODEL_FILE" \
|
|
--served-model-name "$ALIAS" \
|
|
--context "$CONTEXT_SIZE" \
|
|
--speculative dflash2 \
|
|
--dflash-model "$DRAFTER_FILE" \
|
|
--draft-tokens "$DRAFT_TOKENS" \
|
|
--draft-policy "$DRAFT_POLICY" \
|
|
--temperature 0 \
|
|
--think auto \
|
|
--preserve-thinking on
|