frank-large: replace stock llama.cpp with gufo (gufo-org/gufo) on Dankclaw

gufo is a vertical inference engine built for exactly this hardware
(Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large
has swapped runtimes seven times in search of a better one. This one
ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on
the host and the payload, port, alias and client contract stay exactly
where they are.

Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1,
built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4
through its RPATH — the opposite of the unsloth prebuilt it replaces,
which carried its own — so two host workarounds are encoded in that
script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set
that shadows AMD's and cannot be overridden with -I in HIP mode, and the
ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose
libstdc++ is actually installed.

Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL
target and the same DFlash2 drafter, greedy, text-only. Speculation
moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy
capped at 7 proposals.

Client-contract deltas, all documented: the chat template is compiled
into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token
failure mode is gone), reasoning_effort now takes the full OpenAI set
(so models.yml's reasoningEffortMap fold is no longer required — kept
anyway, so this commit does not also move every high role from xhigh to
high effort), and there is no KV-quant or prompt-cache-RAM flag.

The test seam now runs the serve script against a stub engine and
asserts on the argv it actually execs, instead of grepping the script's
text. The deploy handler gates on the runtime binary and returns before
it stops the serving unit.

Measured trade, from gufo's own Q8_K_XL tables against llama.cpp
b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero
depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the
8-bit target the crossover is ~32k, and frank-large runs one 262,144
conversation. The Q4_K_XL target shows no such regression.
This commit is contained in:
Frank
2026-09-28 22:08:29 -07:00
parent d29d104f01
commit aaef419a6d
13 changed files with 695 additions and 240 deletions
+14 -6
View File
@@ -190,13 +190,21 @@ The LLM inference fleet on the two hosts. Full index:
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
rollback record)
- [[frank-large-unsloth-qwen38-27b-q8xl]] — the ACTIVE frank-large runbook
since 2026-09-21 (root systemd unit, port 7770): stock llama.cpp
- [[frank-large-gufo-qwen38-27b]] — the ACTIVE frank-large runbook since
2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1,
the Strix Halo inference engine, built from source on Dankclaw and
serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the
z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — one 262,144-token
conversation, adaptive block policy, greedy, text-only, the two
from-source build workarounds, what changed in the client contract, the
measured prefill/decode trade against llama.cpp
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
(unsloth b11030, ROCm gfx1151) serving
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (a 27B dense model) with the z-lab
DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one 262,144-token
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp v22.4.0
template, the 114 GiB GTT memory model, client contract
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter
(depth 4) on the gfx1151 iGPU — one 262,144-token conversation, q8_0 KV
(the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the
114 GiB GTT memory model, client contract
- [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (replaced by
[[frank-large-unsloth-qwen38-27b-q8xl]]; same-day swap): the Kairic Edge
runtime (ciru-ai/ROCmFPX kairic-edge-qwen38-27b-v1.2, TheRock 7.15.0,