frank-large: replace stock llama.cpp with gufo (gufo-org/gufo) on Dankclaw
gufo is a vertical inference engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large has swapped runtimes seven times in search of a better one. This one ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on the host and the payload, port, alias and client contract stay exactly where they are. Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1, built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4 through its RPATH — the opposite of the unsloth prebuilt it replaces, which carried its own — so two host workarounds are encoded in that script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set that shadows AMD's and cannot be overridden with -I in HIP mode, and the ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose libstdc++ is actually installed. Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL target and the same DFlash2 drafter, greedy, text-only. Speculation moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy capped at 7 proposals. Client-contract deltas, all documented: the chat template is compiled into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token failure mode is gone), reasoning_effort now takes the full OpenAI set (so models.yml's reasoningEffortMap fold is no longer required — kept anyway, so this commit does not also move every high role from xhigh to high effort), and there is no KV-quant or prompt-cache-RAM flag. The test seam now runs the serve script against a stub engine and asserts on the argv it actually execs, instead of grepping the script's text. The deploy handler gates on the runtime binary and returns before it stops the serving unit. Measured trade, from gufo's own Q8_K_XL tables against llama.cpp b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the 8-bit target the crossover is ~32k, and frank-large runs one 262,144 conversation. The Q4_K_XL target shows no such regression.
This commit is contained in:
Frank
+213
-86
@@ -1,31 +1,84 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-large root unit on dankclaw.
|
||||
#
|
||||
# The 2026-09-21 cutover moved this service from the Kairic Edge runtime
|
||||
# (ciru-ai/ROCmFPX) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge onto the
|
||||
# stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter
|
||||
# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M), still as a ROOT unit: a
|
||||
# prebuilt runtime has no container image to install and no generated
|
||||
# unit to dodge. This seam pins what a flag list cannot show: the
|
||||
# operator's one-stream / full-window shape, the payload the serve script
|
||||
# refuses to start without, the DFlash2 spec wiring, fp8 KV, greedy
|
||||
# sampling, a deploy handler that gates on the port, and the fact that
|
||||
# the sweep for a stray benchmark run cannot take frank-small down with
|
||||
# it.
|
||||
|
||||
# Since 2026-09-28 this endpoint is served by gufo (gufo-org/gufo), the
|
||||
# Strix Halo inference engine, built from the pinned v0.1.1 tag and serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter. It replaced
|
||||
# the stock llama.cpp prebuilt (unsloth b11030) on the same port, the same
|
||||
# alias and the same weights.
|
||||
#
|
||||
# Most of this seam runs the serve script against a stub engine and asserts
|
||||
# on the argv it actually execs, so what is pinned is the command the
|
||||
# operator gets rather than the text of the script: a tunable renamed,
|
||||
# dropped, or silently defaulted away fails here the way it would fail a
|
||||
# deployment.
|
||||
load helpers
|
||||
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
|
||||
UNIT="$REPO_ROOT/scripts/frank-large.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
|
||||
BUILD="$REPO_ROOT/scripts/build-gufo.sh"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "the frank-large unit, env and serve files exist where the deployer installs them from" {
|
||||
setup() {
|
||||
STAGE="$(mktemp -d)"
|
||||
RUNTIME_DIR="$STAGE/gufo"
|
||||
MODEL_DIR="$STAGE/models"
|
||||
mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR"
|
||||
# A stub engine: the serve script execs it, so $@ is exactly what gufo
|
||||
# would be invoked with.
|
||||
cat > "$RUNTIME_DIR/bin/gufo" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$@"
|
||||
STUB
|
||||
chmod +x "$RUNTIME_DIR/bin/gufo"
|
||||
: > "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
||||
: > "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
STUB_ENV="$STAGE/stub.env"
|
||||
cat > "$STUB_ENV" <<EOF
|
||||
RUNTIME_DIR=$RUNTIME_DIR
|
||||
MODEL_DIR=$MODEL_DIR
|
||||
EOF
|
||||
}
|
||||
|
||||
teardown() {
|
||||
if [ -n "${STAGE:-}" ]; then
|
||||
rm -rf "$STAGE"
|
||||
fi
|
||||
}
|
||||
|
||||
# serve_argv [env-file] — run the serve script and capture the engine's argv,
|
||||
# one token per line, in $output. No argument exercises the built-in defaults
|
||||
# with the stub's paths.
|
||||
serve_argv() {
|
||||
local envfile="${1:-$STUB_ENV}"
|
||||
if [ -n "$envfile" ]; then
|
||||
FRANK_LARGE_ENV_FILE="$envfile" run bash "$SERVE"
|
||||
else
|
||||
run env FRANK_LARGE_ENV_FILE="$STAGE/absent.env" bash "$SERVE"
|
||||
fi
|
||||
}
|
||||
|
||||
# argv_has <token> — exact token match, so --port never satisfies a test
|
||||
# looking for a different flag.
|
||||
argv_has() {
|
||||
grep -qxF -- "$1" <<<"$output"
|
||||
}
|
||||
|
||||
# argv_value <flag> — the token following <flag>, empty if the flag is absent.
|
||||
argv_value() {
|
||||
local i
|
||||
i="$(grep -nx -- "$1" <<<"$output" | cut -d: -f1)"
|
||||
[ -n "$i" ] || return 0
|
||||
sed -n "$((i + 1))p" <<<"$output"
|
||||
}
|
||||
|
||||
@test "the frank-large unit, env, serve and build files exist where the deployer installs them from" {
|
||||
[ -f "$ENV_FILE" ]
|
||||
[ -f "$UNIT" ]
|
||||
[ -x "$SERVE" ]
|
||||
[ -x "$BUILD" ]
|
||||
}
|
||||
|
||||
@test "frank-large is boot-persistent, self-healing and unprivileged" {
|
||||
@@ -33,65 +86,108 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
# The engine pins staging buffers; the memlock budget must be unbounded
|
||||
# or the load fails on the RLIMIT.
|
||||
# The engine pins buffers; the memlock budget must be unbounded or the
|
||||
# load fails on the RLIMIT.
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
# The unit runs the repo's own serve script with the repo's env.
|
||||
grep -q 'ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh' "$UNIT"
|
||||
grep -q 'FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the operator's shape is one stream of the full 262144 window with DFlash2 depth 4" {
|
||||
local ctx slots
|
||||
ctx="$(sed -nE 's/^CONTEXT_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
slots="$(sed -nE 's/^PARALLEL_SLOTS=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
@test "the fleet endpoint and alias survive the cutover" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
# frank-large at :7770, whatever engine answers it.
|
||||
[ "$(argv_value --served-model-name)" = "frank-large" ]
|
||||
[ "$(argv_value --port)" = "7770" ]
|
||||
[ "$(argv_value --host)" = "0.0.0.0" ]
|
||||
}
|
||||
|
||||
@test "the built-in defaults are one session of the full 262144 window with DFlash2" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
# 262144 is the model's native window (the operator's call, 2026-09-21).
|
||||
[ "$ctx" -eq 262144 ]
|
||||
# One stream.
|
||||
[ "$slots" -eq 1 ]
|
||||
# DFlash2 depth 4 (the drafter's block-diffusion depth).
|
||||
grep -q '^SPEC_DRAFT_N_MAX=4$' "$ENV_FILE"
|
||||
[ "$(argv_value --context)" -eq 262144 ]
|
||||
# One session is the single stream every prior frank-large served.
|
||||
[ "$(argv_value --sessions)" -eq 1 ]
|
||||
[ "$(argv_value --speculative)" = "dflash2" ]
|
||||
# A draft ceiling of 0 would silently serve the autoregressive rate.
|
||||
[ "$(argv_value --draft-tokens)" -gt 0 ]
|
||||
}
|
||||
|
||||
@test "the shipped env file, not just the defaults, serves the deployed shape" {
|
||||
# The defaults are a floor; systemd sources the env file. Only the two
|
||||
# on-host paths are redirected at the stub, so every fleet value in the
|
||||
# shipped file is the one under test.
|
||||
sed -e "s#^RUNTIME_DIR=.*#RUNTIME_DIR=$RUNTIME_DIR#" \
|
||||
-e "s#^MODEL_DIR=.*#MODEL_DIR=$MODEL_DIR#" \
|
||||
"$ENV_FILE" > "$STAGE/shipped.env"
|
||||
serve_argv "$STAGE/shipped.env"
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --served-model-name)" = "frank-large" ]
|
||||
[ "$(argv_value --host)" = "0.0.0.0" ]
|
||||
[ "$(argv_value --port)" = "7770" ]
|
||||
[ "$(argv_value --context)" -eq 262144 ]
|
||||
[ "$(argv_value --sessions)" -eq 1 ]
|
||||
[ "$(argv_value --speculative)" = "dflash2" ]
|
||||
[ "$(argv_value --draft-tokens)" -gt 0 ]
|
||||
[ "$(argv_value --draft-policy)" = "adaptive" ]
|
||||
}
|
||||
|
||||
@test "the engine is pointed at the payload that is actually on disk" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ -f "$(argv_value --model)" ]
|
||||
# A drafter path that does not exist would load the target and decode at
|
||||
# the autoregressive rate, which reads as "the new runtime is slow".
|
||||
[ -f "$(argv_value --dflash-model)" ]
|
||||
[ "$(argv_value --model)" != "$(argv_value --dflash-model)" ]
|
||||
}
|
||||
|
||||
@test "frank-large stays text-only" {
|
||||
# The new model is text-only (no vision projector). The fleet's
|
||||
# image path is frank-small; models.yml declares frank-large text-only.
|
||||
! grep -q -- '--mmproj' "$SERVE"
|
||||
# The payload has no mmproj, so an --mmproj would point at nothing. The
|
||||
# fleet's image path is frank-small; models.yml declares frank-large
|
||||
# text-only.
|
||||
serve_argv
|
||||
! argv_has --mmproj
|
||||
grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
}
|
||||
|
||||
@test "the serve script pins the stock llama.cpp runtime and refuses a missing binary" {
|
||||
grep -q 'unsloth-llama-b11030' "$SERVE"
|
||||
grep -q 'llama-server' "$SERVE"
|
||||
# A missing runtime or payload is a deploy error with a pointer, not a
|
||||
# restart loop.
|
||||
grep -q 'runtime not found' "$SERVE"
|
||||
grep -q 'missing asset' "$SERVE"
|
||||
# The Qwen3.8 template keeps prior thinking blocks across turns only
|
||||
# when the kwarg is set.
|
||||
grep -q 'preserve_thinking' "$SERVE"
|
||||
# The chat template file is wired (not just the kwarg) — the 73k-token
|
||||
# fallback incident (2026-09-21) was a missing --chat-template-file.
|
||||
grep -q 'chat-template-file' "$SERVE"
|
||||
@test "the engine is asked for greedy decoding and for preserved prior reasoning" {
|
||||
# Greedy is what this endpoint has always served; a silent switch to
|
||||
# sampled output would be a model-behaviour change disguised as a
|
||||
# runtime swap. Prior thinking blocks must survive across turns or a
|
||||
# multi-turn agent loses its own reasoning between tool calls.
|
||||
serve_argv
|
||||
[ "$(argv_value --temperature)" = "0" ]
|
||||
[ "$(argv_value --preserve-thinking)" = "on" ]
|
||||
}
|
||||
|
||||
@test "the serve script wires the DFlash2 spec method and the drafter" {
|
||||
# The new model uses DFlash2 (block-diffusion drafter) at depth 4.
|
||||
grep -q -- '--spec-type draft-dflash' "$SERVE"
|
||||
grep -q -- '--model-draft' "$SERVE"
|
||||
grep -q -- '--spec-draft-n-max' "$SERVE"
|
||||
# The drafter runs on the iGPU (ROCm0).
|
||||
grep -q -- '--spec-draft-device ROCm0' "$SERVE"
|
||||
@test "the serve script refuses a missing runtime with the build pointer" {
|
||||
rm -f "$RUNTIME_DIR/bin/gufo"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"runtime not found"* ]]
|
||||
[[ "$output" == *"build-gufo.sh"* ]]
|
||||
# A restart loop would be the alternative; nothing may be exec'd.
|
||||
! argv_has serve
|
||||
}
|
||||
|
||||
@test "the serve script uses q8_0 KV and greedy sampling" {
|
||||
# q8_0 KV — the unsloth b11030 build does not support fp8 KV (allowed:
|
||||
# f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the
|
||||
# highest-precision 8-bit option and preserves the ~8 GiB footprint.
|
||||
grep -q -- '--cache-type-k q8_0' "$SERVE"
|
||||
grep -q -- '--cache-type-v q8_0' "$SERVE"
|
||||
# Greedy sampling (DFlash2 needs greedy verification).
|
||||
grep -q -- '--temp 0' "$SERVE"
|
||||
@test "the serve script refuses a missing drafter with the fetch pointer" {
|
||||
rm -f "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"missing asset"* ]]
|
||||
[[ "$output" == *"download-frank-large.sh"* ]]
|
||||
! argv_has serve
|
||||
}
|
||||
|
||||
@test "the serve script refuses a missing target and names the file" {
|
||||
rm -f "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"missing asset"* ]]
|
||||
[[ "$output" == *"Qwen3.8-27B-UD-Q8_K_XL.gguf"* ]]
|
||||
}
|
||||
|
||||
@test "the payload is fetched resumably and verified against declared sizes + sha256" {
|
||||
@@ -103,41 +199,72 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
grep -q 'SHA256 MISMATCH' "$FETCH"
|
||||
}
|
||||
|
||||
@test "the fleet endpoint and alias survive the cutover" {
|
||||
grep -q '^HOST=0.0.0.0$' "$ENV_FILE"
|
||||
grep -q '^PORT=7770$' "$ENV_FILE"
|
||||
grep -q '^MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL$' "$ENV_FILE"
|
||||
grep -q -- '--alias frank-large' "$SERVE"
|
||||
@test "the build script refuses a source tree that is not the pinned commit" {
|
||||
# gufo ships no prebuilt Linux binaries, so the runtime is built here.
|
||||
# A tag that moves under us would silently install different kernels
|
||||
# under the same RUNTIME_DIR, so the pin is enforced, not documented.
|
||||
local src="$STAGE/not-the-pinned-commit"
|
||||
mkdir -p "$src"
|
||||
git -C "$src" init -q
|
||||
git -C "$src" -c user.email=t@t -c user.name=t commit -q --allow-empty -m drift
|
||||
run env GUFO_SRC_DIR="$src" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"GUFO_COMMIT mismatch"* ]]
|
||||
}
|
||||
|
||||
@test "the build script reports a runtime that was never built" {
|
||||
# --verify on a host with no install must say so rather than exit 0 on
|
||||
# an empty check.
|
||||
run env GUFO_SRC_DIR="$STAGE/no-source-tree" GUFO_PREFIX="$STAGE/absent-prefix" bash "$BUILD" --verify
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"no gufo at"* ]]
|
||||
}
|
||||
|
||||
@test "the deploy handler gates on the runtime, the payload and the port" {
|
||||
grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_RUNTIME' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_MODEL' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_DRAFT' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler will not stop a serving endpoint it cannot replace" {
|
||||
# The runtime gate must be live (a real conditional on an executable
|
||||
# path, not a branch that never fires) and it must return BEFORE the
|
||||
# stop, or a host whose gufo was never built takes frank-large down on
|
||||
# its way to reporting the failure.
|
||||
local large gate bail stop
|
||||
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
|
||||
[[ "$large" == *'build-gufo.sh'* ]]
|
||||
gate="$(grep -n 'if \[\[ ! -x "\$LLM_LARGE_RUNTIME" \]\]' <<<"$large" | head -1 | cut -d: -f1)"
|
||||
stop="$(grep -n 'systemctl stop frank-large' <<<"$large" | head -1 | cut -d: -f1)"
|
||||
[ -n "$gate" ]
|
||||
[ -n "$stop" ]
|
||||
bail="$(awk -v g="$gate" 'NR > g && /return 1/ { print NR; exit }' <<<"$large")"
|
||||
[ -n "$bail" ]
|
||||
[ "$bail" -lt "$stop" ]
|
||||
}
|
||||
|
||||
@test "the stray-run sweep cannot take frank-small down" {
|
||||
# frank-small on this host serves a different GGUF. The sweep must match
|
||||
# the model path, not a binary name, or a frank-large deploy restarts the
|
||||
# 3090 endpoint.
|
||||
grep -q "pkill -f 'Qwen3.8-27B-UD-Q8_K_XL'" "$REMOTE_DEPLOY"
|
||||
local large
|
||||
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
|
||||
! grep -q 'pkill -x llama-server' <<<"$large"
|
||||
! grep -q 'pkill -x gufo' <<<"$large"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the halogen quadlet without deleting it" {
|
||||
grep -q 'retire_halogen_large_quadlet' "$REMOTE_DEPLOY"
|
||||
grep -q 'HOME/.AGENTTRASH' "$REMOTE_DEPLOY"
|
||||
! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?.*frank-large' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler installs the root unit, gates the port and refuses a missing payload" {
|
||||
grep -q 'relink_unit frank-large' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-large.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_MODEL' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_LARGE_DRAFT' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the stray-run sweep cannot take frank-small down" {
|
||||
# frank-small on this host is ALSO a llama-server. The sweep must match
|
||||
# the model path, not the binary name, or a frank-large deploy restarts
|
||||
# the 3090 endpoint.
|
||||
grep -q "pkill -f 'Qwen3.8-27B-UD-Q8_K_XL'" "$REMOTE_DEPLOY"
|
||||
# ...and only inside the large handler: extract it and check there is no
|
||||
# bare binary-name pkill in it.
|
||||
local large
|
||||
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
|
||||
! grep -q 'pkill -x llama-server' <<<"$large"
|
||||
}
|
||||
|
||||
@test "the halogen quadlet is archived, not live" {
|
||||
# Nothing live ships for this endpoint as a container any more; the
|
||||
# archived copy is host state in an ignored .AGENTTRASH, not a tracked
|
||||
# file.
|
||||
[ ! -e "$REPO_ROOT/dankclaw/frank-large/quadlet/frank-large.container" ]
|
||||
[ -f "$REPO_ROOT/dankclaw/.AGENTTRASH/frank-large.container.halogen" ]
|
||||
! grep -rq 'Image=.*halogen' "$REPO_ROOT/dankclaw"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user