serverAppConfigs/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)	The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the one-session / full-window shape, what changed in the client contract versus llama.cpp, the measured prefill/decode trade, and the failure modes worth knowing.	llm gufo unsloth dankclaw frank-large strix-halo gfx1151 q8 dflash2 rocm	2026-09-28

frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2)

ACTIVE since 2026-09-28. Replaces frank-large-unsloth-qwen38-27b-q8xl (stock llama.cpp, the unsloth b11030 prebuilt), which had replaced frank-large-kairic-edge-qwen38-27b. Same host, same port (7770), same alias (frank-large), same weights.

Identity

• Endpoint: frank-large at 192.168.0.143:7770 (Dankclaw).
• Runtime: gufo — "the Strix Halo inference engine", a vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151, unified memory). Pinned at v0.1.1 (commit b0f8673), built from source on the host to /home/dank/GitPackages/gufo-v0.1.1/.
• Service shape: root systemd unit scripts/frank-large.service (User=dank, LimitMEMLOCK=infinity), launched by scripts/serve-frank-large.sh with tunables from scripts/frank-large.env.

gufo ships no prebuilt Linux binaries (the v0.1.0/v0.1.1 release assets are empty) and no Nix on this host, so the runtime is a source build. It is also young: v0.1.0 landed 2026-09-28, the same day as this cutover.

Why this engine

gufo's own measurement of the two engines on the same Qwen3.8-27B targets (Strix Halo, 628ed18e vs llama.cpp b11069, greedy, DFlash2):

Q8_K_XL, DFlash2	gufo	llama.cpp	delta
prefill, depth 0	457.05 t/s	333.85 t/s	+36.9 %
prefill, depth 131,072	237.24 t/s	131.80 t/s	+80.0 %
decode, depth 0 (mixed)	16.08 t/s	15.00 t/s	+7.2 %
decode, depth 32,768 (mixed)	14.41 t/s	14.44 t/s	−0.2 %
decode, depth 65,536 (mixed)	10.87 t/s	12.62 t/s	−13.9 %
decode, depth 131,072 (mixed)	8.87 t/s	11.62 t/s	−23.7 %
cold load to HTTP-ready	8.17 s	36.47 s	4.5× faster
peak memory @ 262,144 ctx	50.58 GiB	48.60 GiB	+4 %

Read that honestly: prefill, load time and short-context decode improve; deep-context decode on the 8-bit target gets worse. The crossover on Q8 is around 32k of cached depth. frank-large runs one 262,144-token conversation, so a long agent session spends its life on the losing side of that line while a short turn wins. The Q4_K_XL target does not show the regression (gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to revisit the quantization separately from the runtime.

Build

scripts/build-gufo.sh reproduces the runtime end to end. It clones the pinned tag, configures the release preset (HIP, --offload-arch=gfx1151), builds, installs, and finishes with gufo diagnose. Two host facts it encodes, both found the hard way on 2026-09-28:

• Ubuntu's librocwmma-dev must not be installed. It ships a truncated rocWMMA header set (the public headers without rocwmma/internal/) into /usr/include, where it shadows AMD's complete copy under /opt/rocm/include. The first build died on 'internal/accessors.hpp' file not found from /usr/include/rocwmma/rocwmma.hpp, and a -I cannot fix it: in HIP mode the host compilation pass searches the system directories first. The package has no reverse dependencies; the script removes it.
• --gcc-install-dir must point at GCC 15. Ubuntu 26.04 has /usr/include/c++/16, and the ROCm clang picks the newest tree, whose <atomic_wait.h> references std::__detail::__notify_impl at a symbol version the installed libstdc++.so.6 (GLIBCXX_3.4.35) does not export. The link then fails with undefined symbol. Pinning the headers to the GCC the runtime library came from fixes it.

Verify an installed runtime with bash scripts/build-gufo.sh --verify, or gufo diagnose directly — it reports [PASS] plus the detected architecture, driver and queue budget.

Shape

• Model: unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL — 27B dense, 29.30 GiB. Unchanged by the cutover, still fetched and sha256-verified by scripts/download-frank-large.sh.
• Context: one session, full 262,144 tokens (the model's native window), SESSIONS=1. The fleet's single stream is unchanged.
• Spec method: DFlash2, gufo's adaptive block policy (--draft-policy adaptive, --draft-tokens 7). The drafter is the same z-lab Qwen3.8-27B-DFlash2-Q4_K_M (1.07 GiB).
• Sampling: greedy (--temperature 0), as every prior frank-large served. gufo's per-model recommendation for Qwen3.8 thinking is sampled (1.0 / 0.95 / top-k 20) — that is a separate decision, deliberately not bundled into a runtime swap.
• Memory: gufo's own table puts the 8-bit target at 50.58 GiB peak with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.

Measured at the 2026-09-28 cutover

Live on Dankclaw after deploy.sh dankclaw --service frank-large:

• 13 s from load_started to listening (target 8 s, drafter 2 s, session 3 s), against a 20-minute LLM_WAIT budget — a redeploy no longer sits in a load wait.
• RSS 30.6 GiB, VSZ 87.3 GiB; host used 81 GiB of 121 GiB, i.e. the same order as the llama.cpp deployment it replaced.
• A short-prompt live request returned Server-Timing: ttft;dur=549.8, inter_token;dur=24.2, max_inter_token;dur=166.9 (≈41 tok/s). Short-prompt decode is gufo's winning regime; see the depth table above before quoting it as the endpoint's number.
• Contract battery, all green: /health 200, /ready 200, /v1/models → frank-large with context_length 262144, a chat completion carrying both content and reasoning_content, a tool_calls response with finish_reason: tool_calls, and SSE streaming.

Client contract — what actually changed

Everything the fleet depended on still holds: port 7770, alias frank-large, reasoning_content, tool_calls, streaming SSE, and text-only (gufo auto-discovers a mmproj beside the model; this payload has none).

The differences worth knowing:

• No custom chat template. gufo compiles the official Qwen Jinja into the binary and takes no --chat-template-file. The Qwen-Sharp v22.4.0 file this endpoint used with llama.cpp is no longer in the command line — the 2026-09-21 "73,600-token prompt" incident cannot recur, because there is no fallback template to land in. Thinking defaults to on at xhigh with prior reasoning preserved (--think auto --preserve-thinking on, spelled out in the serve script).
• reasoning_effort accepts the full OpenAI set (none, minimal, low, medium, high, xhigh, max). Qwen-Sharp rejected anything outside {xhigh, medium, low} with an HTTP 500, which is why models.yml still carries a reasoningEffortMap fold for this model. The fold is kept on purpose: removing it would silently move every high role from xhigh to high effort in the same commit as the runtime swap. Drop it as its own change once the effort curve is a decision rather than a side effect.
• No KV-cache quantization flag. gufo picks its own KV dtype; there is no --cache-type-k. The old CACHE_RAM=8192 tunable is gone with it. Prompt reuse is on by default, and --cache-disk (restart-safe) is opt-in and currently off.
• Health/readiness split: GET /health is process liveness and is what the deployer gates on; GET /ready is model/backend readiness. GET /metrics is Prometheus-format.
• Long bodies: --max-request-bytes defaults to 8 MiB and unknown request fields are rejected rather than ignored.

Failure modes worth knowing

• A missing runtime is a deploy error, not a restart loop. The handler gates on LLM_LARGE_RUNTIME before it stops anything, and the serve script exits with a build-gufo.sh pointer rather than exec'ing.
• Host memory is the ceiling, and it is shared. frank-large and ComfyUI both hold gfx1151 compute queues. gufo reads /sys/class/kfd/kfd/proc/*/queues/*/type and caps itself (GPU_MAX_HW_QUEUES, 2 resident for serve llm) so a full device time-slices instead of failing; four HIP processes is the device ceiling.
• Load is seconds, not minutes (8 s measured cold), so a redeploy no longer spends 20 minutes in LLM_WAIT.
• A v0.1.x runtime is a moving target. Rebuilds are not automatic — RUNTIME_DIR names a version, and the deployer will not silently substitute another.

Rollback

frank-large-unsloth-qwen38-27b-q8xl — the unsloth b11030 prebuilt is still at /home/dank/GitPackages/unsloth-llama-b11030/ and the payload is the same directory the current runtime reads. Roll back by reverting RUNTIME_DIR (and the gufo-only tunables) in scripts/frank-large.env, restoring the serve script's llama.cpp command, and redeploying.

Citations

• Runtime: https://github.com/gufo-org/gufo (pinned v0.1.1, MIT)
• Measured tables: docs/models/qwen3.8-27b/BENCHMARKS.md in that repo
• API contract: docs/SERVER.md; flags: docs/CLI.md
• Build: scripts/build-gufo.sh; serve: scripts/serve-frank-large.sh; tunables: scripts/frank-large.env
• Payload: scripts/download-frank-large.sh
• Predecessor: frank-large-unsloth-qwen38-27b-q8xl