frank-large: replace stock llama.cpp with gufo (gufo-org/gufo) on Dankclaw
gufo is a vertical inference engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large has swapped runtimes seven times in search of a better one. This one ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on the host and the payload, port, alias and client contract stay exactly where they are. Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1, built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4 through its RPATH — the opposite of the unsloth prebuilt it replaces, which carried its own — so two host workarounds are encoded in that script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set that shadows AMD's and cannot be overridden with -I in HIP mode, and the ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose libstdc++ is actually installed. Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL target and the same DFlash2 drafter, greedy, text-only. Speculation moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy capped at 7 proposals. Client-contract deltas, all documented: the chat template is compiled into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token failure mode is gone), reasoning_effort now takes the full OpenAI set (so models.yml's reasoningEffortMap fold is no longer required — kept anyway, so this commit does not also move every high role from xhigh to high effort), and there is no KV-quant or prompt-cache-RAM flag. The test seam now runs the serve script against a stub engine and asserts on the argv it actually execs, instead of grepping the script's text. The deploy handler gates on the runtime binary and returns before it stops the serving unit. Measured trade, from gufo's own Q8_K_XL tables against llama.cpp b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the 8-bit target the crossover is ~32k, and frank-large runs one 262,144 conversation. The Q4_K_XL target shows no such regression.
This commit is contained in:
Frank
Executable
+96
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the gufo runtime that serves frank-large on Dankclaw.
|
||||
#
|
||||
# bash scripts/build-gufo.sh # build + install the pinned tag
|
||||
# bash scripts/build-gufo.sh --verify # only check the installed tree
|
||||
#
|
||||
# gufo (gufo-org/gufo) is the Strix Halo inference engine: it targets
|
||||
# gfx1151 specifically and has no prebuilt Linux release binaries, so the
|
||||
# runtime is built from the pinned git tag and installed to
|
||||
# /home/dank/GitPackages/gufo-<tag>/. Everything below is what Dankclaw
|
||||
# needs on top of AMD's Linux instructions.
|
||||
#
|
||||
# The host already carries the ROCm 7.2.4 stack (/opt/rocm -> /etc/
|
||||
# alternatives/rocm) with the HIP dev packages, so this script installs
|
||||
# only the ordinary system libraries gufo links against.
|
||||
#
|
||||
# Two host facts this build depends on, both discovered on 2026-09-28:
|
||||
#
|
||||
# 1. Ubuntu's librocwmma-dev ships a TRUNCATED rocWMMA header set (the
|
||||
# public headers without rocwmma/internal/) into /usr/include, where
|
||||
# it shadows AMD's complete copy under /opt/rocm/include. The first
|
||||
# build died on "'internal/accessors.hpp' file not found" from
|
||||
# /usr/include/rocwmma/rocwmma.hpp. A -I cannot fix it: in HIP mode
|
||||
# the host compilation pass searches the system directories first.
|
||||
# The package has no reverse dependencies, so it is removed.
|
||||
# 2. The ROCm clang picks the NEWEST /usr/include/c++ tree, which on
|
||||
# Ubuntu 26.04 is GCC 16's. Its <atomic_wait.h> emits a symbol
|
||||
# reference the installed libstdc++ (GLIBCXX_3.4.35, GCC 15) does not
|
||||
# export, and the link fails with "undefined symbol:
|
||||
# std::__detail::__notify_impl". --gcc-install-dir pins the headers to
|
||||
# the same GCC the runtime library came from.
|
||||
set -euo pipefail
|
||||
|
||||
GUFO_TAG="${GUFO_TAG:-v0.1.1}"
|
||||
GUFO_REPO="${GUFO_REPO:-https://github.com/gufo-org/gufo}"
|
||||
# The commit the tag resolved to when this runtime was qualified. A tag that
|
||||
# moves is an unreproducible runtime, so the pin is checked, not assumed.
|
||||
GUFO_COMMIT="${GUFO_COMMIT:-b0f8673470a3dcc2b5e9c9546c48a40b82b5ff31}"
|
||||
SRC_DIR="${GUFO_SRC_DIR:-/home/dank/GitPackages/gufo-src}"
|
||||
PREFIX="${GUFO_PREFIX:-/home/dank/GitPackages/gufo-${GUFO_TAG}}"
|
||||
JOBS="${JOBS:-$(nproc)}"
|
||||
# The GCC toolchain that owns the installed libstdc++ on this host.
|
||||
GCC_DIR="${GCC_DIR:-/usr/lib/gcc/x86_64-linux-gnu/15}"
|
||||
|
||||
# check_pin — refuse a source tree that is not the qualified commit.
|
||||
check_pin() {
|
||||
local commit
|
||||
commit="$(git -C "$SRC_DIR" rev-parse HEAD 2>/dev/null || true)"
|
||||
if [[ -n "$commit" && "$commit" != "$GUFO_COMMIT" ]]; then
|
||||
echo "GUFO_COMMIT mismatch: $SRC_DIR is at $commit, pinned $GUFO_COMMIT" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--verify" ]]; then
|
||||
# The pin first: a moved tag is the more urgent thing to hear about, and
|
||||
# it is the check that must not be masked by a missing install.
|
||||
check_pin
|
||||
[[ -x "$PREFIX/bin/gufo" ]] || { echo "no gufo at $PREFIX/bin/gufo" >&2; exit 1; }
|
||||
"$PREFIX/bin/gufo" diagnose | tail -n 5
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "== system libraries gufo links against =="
|
||||
sudo -n apt-get install -y -qq --no-install-recommends \
|
||||
build-essential cmake ninja-build pkg-config \
|
||||
libicu-dev libcurl4-openssl-dev libssl-dev libpng-dev libjpeg-dev ffmpeg
|
||||
|
||||
echo "== dropping Ubuntu's truncated rocWMMA headers =="
|
||||
if dpkg -s librocwmma-dev >/dev/null 2>&1; then
|
||||
sudo -n apt-get remove -y -qq librocwmma-dev
|
||||
fi
|
||||
|
||||
echo "== source: $GUFO_REPO @ $GUFO_TAG =="
|
||||
if [[ ! -d "$SRC_DIR/.git" ]]; then
|
||||
mkdir -p "$(dirname "$SRC_DIR")"
|
||||
git clone --depth 1 --branch "$GUFO_TAG" "$GUFO_REPO" "$SRC_DIR"
|
||||
fi
|
||||
git -C "$SRC_DIR" fetch --depth 1 origin "refs/tags/$GUFO_TAG"
|
||||
git -C "$SRC_DIR" checkout -q --detach "refs/tags/$GUFO_TAG"
|
||||
echo " $(git -C "$SRC_DIR" rev-parse HEAD)"
|
||||
check_pin
|
||||
|
||||
echo "== configure + build (release, HIP gfx1151) =="
|
||||
cd "$SRC_DIR"
|
||||
toolchain="--gcc-install-dir=$GCC_DIR"
|
||||
cmake --preset release \
|
||||
-DCMAKE_INSTALL_PREFIX="$PREFIX" \
|
||||
-DCMAKE_HIP_FLAGS="$toolchain" \
|
||||
-DCMAKE_EXE_LINKER_FLAGS="$toolchain"
|
||||
cmake --build --preset release --parallel "$JOBS"
|
||||
cmake --install build/release
|
||||
|
||||
echo "== verify =="
|
||||
"$PREFIX/bin/gufo" diagnose
|
||||
echo "frank-large runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)"
|
||||
+29
-15
@@ -1077,18 +1077,25 @@ deploy_anubis() {
|
||||
wait_container_healthy anubis 60 || return 1
|
||||
}
|
||||
|
||||
# frank-large is the stock llama.cpp prebuilt (unsloth b11030, ROCm
|
||||
# gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2
|
||||
# drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU,
|
||||
# as a ROOT unit since the 2026-09-19 cutover: the runtime is a prebuilt
|
||||
# binary, so there is no container image to install and no generated unit
|
||||
# to dodge. relink_unit points /etc/systemd/system/frank-large.service at
|
||||
# the repo file and boot persistence is that unit's [Install] section.
|
||||
# frank-large is gufo (gufo-org/gufo), the Strix Halo inference engine built
|
||||
# from the pinned v0.1.1 tag by scripts/build-gufo.sh, serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter
|
||||
# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU. It is a
|
||||
# ROOT unit: the runtime is a native binary linked against the host ROCm,
|
||||
# so there is no container image to install and no generated unit to dodge.
|
||||
# relink_unit points /etc/systemd/system/frank-large.service at the repo
|
||||
# file and boot persistence is that unit's [Install] section.
|
||||
#
|
||||
# It replaces the Kairic Edge runtime (ciru-ai/ROCmFPX), which had
|
||||
# replaced the unsloth llama.cpp prebuilt serving
|
||||
# unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS. The port gate is the same as
|
||||
# every cutover since 2026-09-16: nothing starts until 7770 is released.
|
||||
# The runtime gate matters as much as the payload gate: gufo is built from
|
||||
# source on this host (there is no prebuilt release), so a machine whose
|
||||
# /home/dank/GitPackages tree was never built has a unit that can never
|
||||
# become healthy, and a green deploy would leave the endpoint dark.
|
||||
#
|
||||
# It replaces the stock llama.cpp prebuilt (unsloth b11030), which had
|
||||
# replaced the Kairic Edge runtime (ciru-ai/ROCmFPX). The port gate is the
|
||||
# same as every cutover since 2026-09-16: nothing starts until 7770 is
|
||||
# released.
|
||||
LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}"
|
||||
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf}"
|
||||
LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
|
||||
@@ -1126,6 +1133,11 @@ deploy_llm_large() {
|
||||
# Refuse rather than restart-loop. A unit whose runtime or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the deploy log to say why.
|
||||
if [[ ! -x "$LLM_LARGE_RUNTIME" ]]; then
|
||||
echo "ERROR: gufo runtime absent at $LLM_LARGE_RUNTIME" >&2
|
||||
echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
local missing=0 f
|
||||
for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
@@ -1140,10 +1152,12 @@ deploy_llm_large() {
|
||||
# On a re-deploy the predecessor is THIS unit: stop it before waiting on
|
||||
# the port, or the wait deadlocks on the server it just stopped serving.
|
||||
# Then sweep a stray manual run of the SAME model -- that is how a bench
|
||||
# harness leaves the port busy. Match the model path, not the binary name:
|
||||
# frank-small on this host is also a llama-server and must not be killed.
|
||||
# The pattern never appears in the deploying shell's own command line, so
|
||||
# -f is safe here (it is not in `bash deploy-remote.sh`).
|
||||
# harness leaves the port busy, and the pattern matches the gufo command
|
||||
# line because it carries the model path. Match the model path, not a
|
||||
# binary name: frank-small on this host serves a different GGUF over
|
||||
# llama-server and must not be killed. The pattern never appears in the
|
||||
# deploying shell's own command line, so -f is safe here (it is not in
|
||||
# `bash deploy-remote.sh`).
|
||||
run_privileged systemctl stop frank-large.service 2>/dev/null || true
|
||||
pkill -f 'Qwen3.8-27B-UD-Q8_K_XL' 2>/dev/null || true
|
||||
# Nothing may start until the predecessor has released the port. On
|
||||
|
||||
+18
-18
@@ -1,6 +1,5 @@
|
||||
# frank-large — stock llama.cpp (mainline, the unsloth b11030 prebuilt)
|
||||
# serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU
|
||||
# (gfx1151).
|
||||
# frank-large — gufo (gufo-org/gufo) serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU (gfx1151).
|
||||
#
|
||||
# These are the tunables read by scripts/serve-frank-large.sh. The flag
|
||||
# list is owned by the serve script; this file only sets fleet values.
|
||||
@@ -10,27 +9,28 @@
|
||||
# exception because the serve script itself consumes it for --host.
|
||||
|
||||
# The payload (scripts/download-frank-large.sh): the serve script refuses to
|
||||
# start without the main GGUF and the DFlash2 drafter.
|
||||
# start without the main GGUF and the DFlash2 drafter. Both files are the
|
||||
# same ones the llama.cpp deployment used — the cutover changed the RUNTIME,
|
||||
# not the weights.
|
||||
MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
|
||||
|
||||
# The stock llama.cpp runtime (the unsloth b11030 prebuilt).
|
||||
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
|
||||
# The gufo runtime, built from source by scripts/build-gufo.sh.
|
||||
RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1
|
||||
|
||||
# Fleet endpoint: the same port and alias the prior deployments served, so
|
||||
# Fleet endpoint: the same port and alias every prior frank-large served, so
|
||||
# no client, probe or models.yml entry changes address.
|
||||
ALIAS=frank-large
|
||||
HOST=0.0.0.0
|
||||
PORT=7770
|
||||
|
||||
# One stream, full 262144 context (the model's native window).
|
||||
# One session, full 262144 context (the model's native window).
|
||||
CONTEXT_SIZE=262144
|
||||
PARALLEL_SLOTS=1
|
||||
SESSIONS=1
|
||||
|
||||
# DFlash2 spec decode (a block-diffusion drafter) at depth 4. The drafter
|
||||
# has a sliding window of 2048 (small draft KV, ~40 MiB).
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
|
||||
# Prompt-cache budget: 8 GiB. The model is a 27B hybrid (16 full attention
|
||||
# + 48 Gated DeltaNet linear attention); its DeltaNet state is smaller than
|
||||
# a primarily-DeltaNet model, so the fleet's 12.8 GiB DeltaNet floor does
|
||||
# not apply.
|
||||
CACHE_RAM=8192
|
||||
# DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft
|
||||
# block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is
|
||||
# 7, chosen over llama.cpp's fixed depth 4 because gufo's adaptive policy
|
||||
# shortens the block when acceptance drops) and DRAFT_POLICY is the block
|
||||
# length controller.
|
||||
DRAFT_TOKENS=7
|
||||
DRAFT_POLICY=adaptive
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=stock llama.cpp (unsloth b11030, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, fp8 KV, 262144 ctx / 1 slot, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
|
||||
Description=gufo (gufo-org/gufo, ROCm gfx1151) Qwen3.8-27B UD-Q8_K_XL, DFlash2 spec decode, 262144 ctx / 1 session, root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
@@ -1,32 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on
|
||||
# Dankclaw's Strix Halo iGPU (gfx1151), using stock llama.cpp
|
||||
# (mainline, the unsloth b11030 prebuilt).
|
||||
# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on Dankclaw's
|
||||
# Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the inference
|
||||
# engine built specifically for this hardware.
|
||||
#
|
||||
# Runtime: /home/dank/GitPackages/unsloth-llama-b11030/llama-server
|
||||
# (mainline, ROCm gfx1151 prebuilt). The build supports the qwen3_5
|
||||
# architecture and the DFlash2 drafter (llama_model_qwen35,
|
||||
# llama_model_dflash, common_speculative_impl_draft_dflash).
|
||||
# Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source
|
||||
# against the host ROCm by scripts/build-gufo.sh. It replaces the stock
|
||||
# llama.cpp prebuilt (unsloth b11030) that served this endpoint from
|
||||
# 2026-09-21; payload, port and alias are unchanged, so no client moved.
|
||||
#
|
||||
# Shape: one stream, full 262144 context (the model's native window), all
|
||||
# layers on the iGPU (--n-gpu-layers 999). The 27B dense model (31.46 GiB GGUF) + the
|
||||
# DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) + compute buffers fits the
|
||||
# iGPU's GTT aperture (~114 GiB).
|
||||
# Shape: one session, full 262144 context (the model's native window). The
|
||||
# 27B dense weights (29.30 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
|
||||
# compute buffers fit the iGPU's GTT aperture (~114 GiB); gufo's own
|
||||
# Q8/AR table measures 50.58 GiB peak at this context capacity.
|
||||
#
|
||||
# Spec method: DFlash2 (a block-diffusion drafter) at depth 4. The drafter
|
||||
# has a sliding window of 2048 (small draft KV, ~40 MiB).
|
||||
# Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block
|
||||
# policy, capped at DRAFT_TOKENS proposals per step. The drafter is the
|
||||
# z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048
|
||||
# (small draft KV).
|
||||
#
|
||||
# KV: q8_0 quantized (--cache-type-k q8_0 --cache-type-v q8_0). The unsloth b11030
|
||||
# build does not support fp8 KV (allowed: f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/
|
||||
# q5_1); q8_0 is the highest-precision 8-bit option and preserves the ~8 GiB footprint.
|
||||
# Sampling: greedy (--temperature 0). That is what this endpoint has always
|
||||
# served, and it keeps the swap to a pure runtime change; gufo's per-model
|
||||
# recommendation for Qwen3.8 thinking is sampled (1.0 / 0.95 / top-k 20) and
|
||||
# is a separate decision.
|
||||
#
|
||||
# Text-only (no mmproj) — matching the prior frank-large.
|
||||
# Template: gufo compiles the official Qwen Jinja into the binary and takes
|
||||
# no --chat-template-file. Its Qwen27B default is thinking ON at xhigh effort
|
||||
# with prior reasoning preserved, which is the contract the fleet already
|
||||
# depends on, so --think auto + --preserve-thinking on is spelled out here
|
||||
# rather than left implicit.
|
||||
#
|
||||
# The Qwen3.8 family template keeps prior thinking blocks across turns when
|
||||
# preserve_thinking is set; pass --jinja and the kwarg so multi-turn agents
|
||||
# do not lose the trail. The template gates reasoning_effort to {xhigh,
|
||||
# medium, low} — the harness folds its levels onto that set in models.yml
|
||||
# (reasoningEffortMap).
|
||||
# Text-only: gufo auto-discovers a mmproj beside the model and this payload
|
||||
# has none, so the endpoint stays text-only.
|
||||
#
|
||||
# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
|
||||
# it; anything not set there falls through to the defaults below.
|
||||
@@ -41,22 +45,23 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) ------------------
|
||||
: "${RUNTIME_DIR:=/home/dank/GitPackages/unsloth-llama-b11030}"
|
||||
: "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
|
||||
: "${ALIAS:=frank-large}"
|
||||
: "${HOST:=0.0.0.0}"
|
||||
: "${PORT:=7770}"
|
||||
: "${CONTEXT_SIZE:=262144}"
|
||||
: "${PARALLEL_SLOTS:=1}"
|
||||
: "${SPEC_DRAFT_N_MAX:=4}"
|
||||
: "${CACHE_RAM:=8192}"
|
||||
TEMPLATE_FILE="${TEMPLATE_FILE:-$SCRIPT_DIR/qwen-sharp-v22.4.0.jinja}"
|
||||
: "${SESSIONS:=1}"
|
||||
: "${DRAFT_TOKENS:=7}"
|
||||
: "${DRAFT_POLICY:=adaptive}"
|
||||
|
||||
SERVER_BIN="$RUNTIME_DIR/llama-server"
|
||||
GUFO_BIN="$RUNTIME_DIR/bin/gufo"
|
||||
MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
||||
DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
|
||||
if [[ ! -x "$SERVER_BIN" ]]; then
|
||||
echo "runtime not found at $SERVER_BIN" >&2
|
||||
if [[ ! -x "$GUFO_BIN" ]]; then
|
||||
echo "runtime not found at $GUFO_BIN" >&2
|
||||
echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
|
||||
@@ -66,42 +71,28 @@ for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
|
||||
fi
|
||||
done
|
||||
|
||||
# --- env (the unsloth build's ROCm setup) --------------------------------
|
||||
# --- env -------------------------------------------------------------------
|
||||
# gufo links against the host ROCm through its RPATH (/opt/rocm ->
|
||||
# /etc/alternatives/rocm), so no LD_LIBRARY_PATH is needed. The device is
|
||||
# already gfx1151 — `gufo diagnose` reports it natively — so no
|
||||
# HSA_OVERRIDE_GFX_VERSION either; the override existed for the prebuilt
|
||||
# llama.cpp runtime and can only misreport the arch here.
|
||||
export HIP_VISIBLE_DEVICES=0
|
||||
export HSA_OVERRIDE_GFX_VERSION=11.5.1
|
||||
export LD_LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/llvm/lib:${LD_LIBRARY_PATH:-}"
|
||||
|
||||
# --- the serve command ---------------------------------------------------
|
||||
ARGS=(
|
||||
--host "$HOST"
|
||||
--port "$PORT"
|
||||
--model "$MODEL_FILE"
|
||||
--alias frank-large
|
||||
--ctx-size "$CONTEXT_SIZE"
|
||||
--parallel "$PARALLEL_SLOTS"
|
||||
--n-gpu-layers 999
|
||||
--jinja
|
||||
--metrics
|
||||
--chat-template-file "$TEMPLATE_FILE"
|
||||
--chat-template-kwargs '{"preserve_thinking":true}'
|
||||
--cache-type-k q8_0
|
||||
--cache-type-v q8_0
|
||||
--cache-ram "$CACHE_RAM"
|
||||
# DFlash2 spec decode (a block-diffusion drafter)
|
||||
--spec-type draft-dflash
|
||||
--model-draft "$DRAFTER_FILE"
|
||||
--spec-draft-n-max "$SPEC_DRAFT_N_MAX"
|
||||
--spec-draft-n-min 0
|
||||
--spec-draft-device ROCm0
|
||||
# Greedy sampling (the DFlash2 drafter needs greedy verification)
|
||||
--temp 0
|
||||
--top-p 1.0
|
||||
--top-k 0
|
||||
--min-p 0.0
|
||||
# Reasoning (thoughts land in reasoning_content)
|
||||
--reasoning auto
|
||||
--reasoning-format deepseek
|
||||
--reasoning-budget -1
|
||||
)
|
||||
|
||||
exec "$SERVER_BIN" "${ARGS[@]}"
|
||||
# Server options precede the subcommand, model options follow it.
|
||||
exec "$GUFO_BIN" serve \
|
||||
--host "$HOST" \
|
||||
--port "$PORT" \
|
||||
--sessions "$SESSIONS" \
|
||||
llm \
|
||||
--model "$MODEL_FILE" \
|
||||
--served-model-name "$ALIAS" \
|
||||
--context "$CONTEXT_SIZE" \
|
||||
--speculative dflash2 \
|
||||
--dflash-model "$DRAFTER_FILE" \
|
||||
--draft-tokens "$DRAFT_TOKENS" \
|
||||
--draft-policy "$DRAFT_POLICY" \
|
||||
--temperature 0 \
|
||||
--think auto \
|
||||
--preserve-thinking on
|
||||
|
||||
Reference in New Issue
Block a user