0 Fork 0

Files

T

Frank aaef419a6d frank-large: replace stock llama.cpp with gufo (gufo-org/gufo) on Dankclaw

gufo is a vertical inference engine built for exactly this hardware
(Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large
has swapped runtimes seven times in search of a better one. This one
ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on
the host and the payload, port, alias and client contract stay exactly
where they are.

Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1,
built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4
through its RPATH — the opposite of the unsloth prebuilt it replaces,
which carried its own — so two host workarounds are encoded in that
script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set
that shadows AMD's and cannot be overridden with -I in HIP mode, and the
ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose
libstdc++ is actually installed.

Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL
target and the same DFlash2 drafter, greedy, text-only. Speculation
moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy
capped at 7 proposals.

Client-contract deltas, all documented: the chat template is compiled
into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token
failure mode is gone), reasoning_effort now takes the full OpenAI set
(so models.yml's reasoningEffortMap fold is no longer required — kept
anyway, so this commit does not also move every high role from xhigh to
high effort), and there is no KV-quant or prompt-cache-RAM flag.

The test seam now runs the serve script against a stub engine and
asserts on the argv it actually execs, instead of grepping the script's
text. The deploy handler gates on the runtime binary and returns before
it stops the serving unit.

Measured trade, from gufo's own Q8_K_XL tables against llama.cpp
b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero
depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the
8-bit target the crossover is ~32k, and frank-large runs one 262,144
conversation. The Q4_K_XL target shows no such regression.

2026-09-28 22:08:29 -07:00

8.7 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2)

ACTIVE since 2026-09-28. Replaces frank-large-unsloth-qwen38-27b-q8xl (stock llama.cpp, the unsloth b11030 prebuilt), which had replaced frank-large-kairic-edge-qwen38-27b. Same host, same port (7770), same alias (frank-large), same weights.

Identity

Endpoint: frank-large at 192.168.0.143:7770 (Dankclaw).
Runtime: gufo — "the Strix Halo inference engine", a vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151, unified memory). Pinned at v0.1.1 (commit b0f8673), built from source on the host to /home/dank/GitPackages/gufo-v0.1.1/.
Service shape: root systemd unit scripts/frank-large.service (User=dank, LimitMEMLOCK=infinity), launched by scripts/serve-frank-large.sh with tunables from scripts/frank-large.env.

gufo ships no prebuilt Linux binaries (the v0.1.0/v0.1.1 release assets are empty) and no Nix on this host, so the runtime is a source build. It is also young: v0.1.0 landed 2026-09-28, the same day as this cutover.

Why this engine

gufo's own measurement of the two engines on the same Qwen3.8-27B targets (Strix Halo, 628ed18e vs llama.cpp b11069, greedy, DFlash2):

Q8_K_XL, DFlash2	gufo	llama.cpp	delta
prefill, depth 0	457.05 t/s	333.85 t/s	+36.9 %
prefill, depth 131,072	237.24 t/s	131.80 t/s	+80.0 %
decode, depth 0 (mixed)	16.08 t/s	15.00 t/s	+7.2 %
decode, depth 32,768 (mixed)	14.41 t/s	14.44 t/s	−0.2 %
decode, depth 65,536 (mixed)	10.87 t/s	12.62 t/s	−13.9 %
decode, depth 131,072 (mixed)	8.87 t/s	11.62 t/s	−23.7 %
cold load to HTTP-ready	8.17 s	36.47 s	4.5× faster
peak memory @ 262,144 ctx	50.58 GiB	48.60 GiB	+4 %

Read that honestly: prefill, load time and short-context decode improve; deep-context decode on the 8-bit target gets worse. The crossover on Q8 is around 32k of cached depth. frank-large runs one 262,144-token conversation, so a long agent session spends its life on the losing side of that line while a short turn wins. The Q4_K_XL target does not show the regression (gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to revisit the quantization separately from the runtime.

Build

scripts/build-gufo.sh reproduces the runtime end to end. It clones the pinned tag, configures the release preset (HIP, --offload-arch=gfx1151), builds, installs, and finishes with gufo diagnose. Two host facts it encodes, both found the hard way on 2026-09-28:

Ubuntu's librocwmma-dev must not be installed. It ships a truncated rocWMMA header set (the public headers without rocwmma/internal/) into /usr/include, where it shadows AMD's complete copy under /opt/rocm/include. The first build died on 'internal/accessors.hpp' file not found from /usr/include/rocwmma/rocwmma.hpp, and a -I cannot fix it: in HIP mode the host compilation pass searches the system directories first. The package has no reverse dependencies; the script removes it.
--gcc-install-dir must point at GCC 15. Ubuntu 26.04 has /usr/include/c++/16, and the ROCm clang picks the newest tree, whose <atomic_wait.h> references std::__detail::__notify_impl at a symbol version the installed libstdc++.so.6 (GLIBCXX_3.4.35) does not export. The link then fails with undefined symbol. Pinning the headers to the GCC the runtime library came from fixes it.

Verify an installed runtime with bash scripts/build-gufo.sh --verify, or gufo diagnose directly — it reports [PASS] plus the detected architecture, driver and queue budget.

Shape

Model: unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL — 27B dense, 29.30 GiB. Unchanged by the cutover, still fetched and sha256-verified by scripts/download-frank-large.sh.
Context: one session, full 262,144 tokens (the model's native window), SESSIONS=1. The fleet's single stream is unchanged.
Spec method: DFlash2, gufo's adaptive block policy (--draft-policy adaptive, --draft-tokens 7). The drafter is the same z-lab Qwen3.8-27B-DFlash2-Q4_K_M (1.07 GiB).
Sampling: greedy (--temperature 0), as every prior frank-large served. gufo's per-model recommendation for Qwen3.8 thinking is sampled (1.0 / 0.95 / top-k 20) — that is a separate decision, deliberately not bundled into a runtime swap.
Memory: gufo's own table puts the 8-bit target at 50.58 GiB peak with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.

Client contract — what actually changed

Everything the fleet depended on still holds: port 7770, alias frank-large, reasoning_content, tool_calls, streaming SSE, and text-only (gufo auto-discovers a mmproj beside the model; this payload has none).

The differences worth knowing:

No custom chat template. gufo compiles the official Qwen Jinja into the binary and takes no --chat-template-file. The Qwen-Sharp v22.4.0 file this endpoint used with llama.cpp is no longer in the command line — the 2026-09-21 "73,600-token prompt" incident cannot recur, because there is no fallback template to land in. Thinking defaults to on at xhigh with prior reasoning preserved (--think auto --preserve-thinking on, spelled out in the serve script).
reasoning_effort accepts the full OpenAI set (none, minimal, low, medium, high, xhigh, max). Qwen-Sharp rejected anything outside {xhigh, medium, low} with an HTTP 500, which is why models.yml still carries a reasoningEffortMap fold for this model. The fold is kept on purpose: removing it would silently move every high role from xhigh to high effort in the same commit as the runtime swap. Drop it as its own change once the effort curve is a decision rather than a side effect.
No KV-cache quantization flag. gufo picks its own KV dtype; there is no --cache-type-k. The old CACHE_RAM=8192 tunable is gone with it. Prompt reuse is on by default, and --cache-disk (restart-safe) is opt-in and currently off.
Health/readiness split: GET /health is process liveness and is what the deployer gates on; GET /ready is model/backend readiness. GET /metrics is Prometheus-format.
Long bodies: --max-request-bytes defaults to 8 MiB and unknown request fields are rejected rather than ignored.

Failure modes worth knowing

A missing runtime is a deploy error, not a restart loop. The handler gates on LLM_LARGE_RUNTIME before it stops anything, and the serve script exits with a build-gufo.sh pointer rather than exec'ing.
Host memory is the ceiling, and it is shared. frank-large and ComfyUI both hold gfx1151 compute queues. gufo reads /sys/class/kfd/kfd/proc/*/queues/*/type and caps itself (GPU_MAX_HW_QUEUES, 2 resident for serve llm) so a full device time-slices instead of failing; four HIP processes is the device ceiling.
Load is seconds, not minutes (8 s measured cold), so a redeploy no longer spends 20 minutes in LLM_WAIT.
A v0.1.x runtime is a moving target. Rebuilds are not automatic — RUNTIME_DIR names a version, and the deployer will not silently substitute another.

Rollback

frank-large-unsloth-qwen38-27b-q8xl — the unsloth b11030 prebuilt is still at /home/dank/GitPackages/unsloth-llama-b11030/ and the payload is the same directory the current runtime reads. Roll back by reverting RUNTIME_DIR (and the gufo-only tunables) in scripts/frank-large.env, restoring the serve script's llama.cpp command, and redeploying.

Citations

Runtime: https://github.com/gufo-org/gufo (pinned v0.1.1, MIT)
Measured tables: docs/models/qwen3.8-27b/BENCHMARKS.md in that repo
API contract: docs/SERVER.md; flags: docs/CLI.md
Build: scripts/build-gufo.sh; serve: scripts/serve-frank-large.sh; tunables: scripts/frank-large.env
Payload: scripts/download-frank-large.sh
Predecessor: frank-large-unsloth-qwen38-27b-q8xl

8.7 KiB Raw Blame History Unescape Escape