serverAppConfigs/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)	The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing.	llm gufo swift-1.5 dankclaw frank-large strix-halo gfx1151 q8 dflash2 rocm	2026-09-28

frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2)

ACTIVE since 2026-09-28. Replaces frank-large-unsloth-qwen38-27b-q8xl (stock llama.cpp, the unsloth b11030 prebuilt), which had replaced frank-large-kairic-edge-qwen38-27b. Same host, same port (7770), same alias (frank-large).

The payload also changed on 2026-09-28, later the same day: the target went from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL to ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's UD-Q8_K_XL — an abliterated (uncensored) fine-tune of the same base. See Payload and, for what that costs, the template law.

Identity

• Endpoint: frank-large at 192.168.0.143:7770 (Dankclaw).
• Runtime: gufo — "the Strix Halo inference engine", a vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151, unified memory). Pinned at v0.1.1 (commit b0f8673), built from source on the host to /home/dank/GitPackages/gufo-v0.1.1/.
• Service shape: root systemd unit scripts/frank-large.service (User=dank, LimitMEMLOCK=infinity), launched by scripts/serve-frank-large.sh with tunables from scripts/frank-large.env.

gufo ships no prebuilt Linux binaries (the v0.1.0/v0.1.1 release assets are empty) and no Nix on this host, so the runtime is a source build. It is also young: v0.1.0 landed 2026-09-28, the same day as this cutover.

Why this engine

gufo's own measurement of the two engines on the same Qwen3.8-27B targets (Strix Halo, 628ed18e vs llama.cpp b11069, greedy, DFlash2):

Q8_K_XL, DFlash2	gufo	llama.cpp	delta
prefill, depth 0	457.05 t/s	333.85 t/s	+36.9 %
prefill, depth 131,072	237.24 t/s	131.80 t/s	+80.0 %
decode, depth 0 (mixed)	16.08 t/s	15.00 t/s	+7.2 %
decode, depth 32,768 (mixed)	14.41 t/s	14.44 t/s	−0.2 %
decode, depth 65,536 (mixed)	10.87 t/s	12.62 t/s	−13.9 %
decode, depth 131,072 (mixed)	8.87 t/s	11.62 t/s	−23.7 %
cold load to HTTP-ready	8.17 s	36.47 s	4.5× faster
peak memory @ 262,144 ctx	50.58 GiB	48.60 GiB	+4 %

Read that honestly: prefill, load time and short-context decode improve; deep-context decode on the 8-bit target gets worse. The crossover on Q8 is around 32k of cached depth. frank-large runs one 262,144-token conversation, so a long agent session spends its life on the losing side of that line while a short turn wins. The Q4_K_XL target does not show the regression (gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to revisit the quantization separately from the runtime.

Build

scripts/build-gufo.sh reproduces the runtime end to end. It clones the pinned tag, configures the release preset (HIP, --offload-arch=gfx1151), builds, installs, and finishes with gufo diagnose. Two host facts it encodes, both found the hard way on 2026-09-28:

• Ubuntu's librocwmma-dev must not be installed. It ships a truncated rocWMMA header set (the public headers without rocwmma/internal/) into /usr/include, where it shadows AMD's complete copy under /opt/rocm/include. The first build died on 'internal/accessors.hpp' file not found from /usr/include/rocwmma/rocwmma.hpp, and a -I cannot fix it: in HIP mode the host compilation pass searches the system directories first. The package has no reverse dependencies; the script removes it.
• --gcc-install-dir must point at GCC 15. Ubuntu 26.04 has /usr/include/c++/16, and the ROCm clang picks the newest tree, whose <atomic_wait.h> references std::__detail::__notify_impl at a symbol version the installed libstdc++.so.6 (GLIBCXX_3.4.35) does not export. The link then fails with undefined symbol. Pinning the headers to the GCC the runtime library came from fixes it.

Verify an installed runtime with bash scripts/build-gufo.sh --verify, or gufo diagnose directly — it reports [PASS] plus the detected architecture, driver and queue budget.

Shape

Payload

• Model: ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF, file UD-Q8_K_XL — 27B dense, 31,457,990,272 bytes, fetched and sha256-verified (9076f99a…, the Hub LFS oid) by scripts/download-frank-large-swift.sh. It is an abliterated (uncensored) fine-tune of Qwen3.8-27B. The repository also publishes an mmproj-BF16.gguf, which is not used: this endpoint is text-only.
• Licence: the repo declares other — the SWIFT OPEN LICENSE 1.0, which is not an OSI licence. That is a real constraint on redistribution, noted here so nobody discovers it later. It is fine for a private fleet.
• Replaced: unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL, still on disk and still the rollback target for the payload.

The two targets are structurally interchangeable for this engine, which is why the swap needed no engine change. Header-diffed directly:

	unsloth	Swift-1.5
general.architecture	qwen35	qwen35
tensors (count, names, order)	866	866, identical
per-tensor ggml_type	Q8_0×453, F32×360, BF16×53	identical
per-tensor dims	—	0 mismatches
MTP head	4 tensors at blk.64.nextn.*	identical
geometry	65 blk, 5120 embd, 17408 FFN, 24/4 heads	identical

The two metadata differences that look alarming are both no-ops, and both were confirmed against gufo's source rather than assumed:

• general.file_type is 15 vs 7. gufo's gguf_reader.cpp states it is "deliberately not consulted… The Unsloth UD-*_K_XL artifacts this repo targets carry a stale one — UD-Q8_K_XL records ftype 15 (Q4_K_M) … The tensor table always is." The file gufo already served lies about itself here, so gufo reads the tensor table.
• Swift adds qwen35.attention.recurrent_layers. gufo never reads that key anywhere in src/; layer kinds come from full_attention_interval, which both files carry as 4 and which agrees with it.

Serving shape

• Context: one session, full 262,144 tokens (the model's native window), SESSIONS=1. The fleet's single stream is unchanged.
• Spec method: DFlash2, gufo's adaptive block policy (--draft-policy adaptive, --draft-tokens 7). The drafter is the unchanged z-lab Qwen3.8-27B-DFlash2-Q4_K_M (1.07 GiB) — it did not move with the target, which is why DRAFTER_DIR is a separate tunable from MODEL_DIR and why the deployer's stray-run sweep keys on the drafter's filename rather than the target's.
• Sampling: greedy (--temperature 0) server-side, as every prior frank-large served. The harness overrides per request via extraBody (1.0 / 0.95 / top-k 20).
• Memory: gufo's own table puts the 8-bit target at 50.58 GiB peak with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.

Measured at the 2026-09-28 cutover

Live on Dankclaw. The target numbers are the Swift-1.5 load test (a standalone run under the production flags, service stopped first); the engine numbers are from the gufo cutover earlier the same day and are unchanged by the payload swap.

• 9.2 s load_started → load_completed for the Swift target, against a 20-minute LLM_WAIT budget. The unsloth target loaded in 13 s.
• 66.5 GiB of the 116.7 GiB iGPU device at context capacity 262,144; host RSS 30.6 GiB. No loader warnings, no skipped sections.
• Contract battery, all green on the Swift target: /v1/models → context_length 262144, a chat completion carrying both content and reasoning_content, a tool_calls response with finish_reason: tool_calls, SSE streaming, and a two-turn tool loop (tool call → tool result → follow-up) with prior reasoning preserved. Arithmetic spot-check: "17*23" → 391.
• Behavioural spot-check that this is the abliterated model: asked for a two-sentence lockpicking scene, it wrote one without hedging or refusing.
• From the gufo cutover, on the unsloth target: a short-prompt live request returned Server-Timing: ttft;dur=549.8, inter_token;dur=24.2 (≈41 tok/s). Short-prompt decode is gufo's winning regime; see the depth table above before quoting it as the endpoint's number.

The template law: Qwen-Sharp is gone

scripts/qwen-sharp-v22.4.0.jinja is the fleet's chat template — it is wired into frank-primary as R9V_CHAT_TEMPLATE_FILE and it was frank-large's --chat-template-file under llama.cpp. frank-large can no longer use it, and the reason is stronger than "gufo lacks the flag":

gufo has no Jinja engine at all. src/models/qwen/chat_template.cpp is 587 lines of hand-written C++ that appends strings to build ChatML. There is no --chat-template-file, no environment override, and no template body anywhere in the binary to point at. Qwen-Sharp is 453 lines of Jinja2 using namespace(), macros, raise_exception and image/video branches — supplying it would mean reimplementing it in C++, not handing gufo a file.

Given the choice on 2026-09-28, the decision was to ship on gufo's stock Jinja and accept the loss. What that gives up, concretely:

Qwen-Sharp feature	On gufo
terse system-prompt injection (the headline behaviour)	gone
in-band <|think_off|> / <|think_xhigh|> control	gone
tool_call_format json | xml switch	gone
max_tool_arg_chars / max_tool_response_chars truncation	gone
add_vision_id → Picture N: prefixes	gone
effort instruction strings in the system prompt	kept (gufo has its own, same text)
preserve_thinking	kept (--preserve-thinking on)

The terse injection is the one that matters most here, and it is worth being blunt about the combination: this endpoint now runs an abliterated model without the prompt that tells it to stop waffling. If verbosity shows up, the cheapest fix is to send the terse text as an ordinary system message from the client (models.yml / the harness) rather than through a template — that recovers the behaviour without forking the renderer. The full-fidelity options are reverting to llama.cpp or porting Qwen-Sharp into the C++ renderer; both are worse trades today, and the second is the same class of change that produced the 2026-09-21 "73,600-token prompt" incident.

Client contract — what actually changed

Everything the fleet depended on still holds: port 7770, alias frank-large, reasoning_content, tool_calls, streaming SSE, and text-only (gufo auto-discovers a mmproj beside the model; this payload has none).

The differences worth knowing:

• No custom chat template — see The template law above, which is the most consequential change on this endpoint. Thinking defaults to on at xhigh with prior reasoning preserved (--think auto --preserve-thinking on, spelled out in the serve script).

• reasoning_effort accepts the full OpenAI set (none, minimal, low, medium, high, xhigh, max), all verified 200 against the live endpoint. Qwen-Sharp rejected anything outside {xhigh, medium, low} with an HTTP 500, which is why models.yml still carries a reasoningEffortMap fold for this model. The fold is kept on purpose: removing it would silently move every high role from xhigh to high effort. Drop it as its own change.

• Every field the harness sends is accepted. top_k, min_p, presence_penalty, repetition_penalty and preserve_thinking are all outside gufo's documented request subset, and all returned 200. They are applied from models.yml's extraBody unchanged.

• No KV-cache quantization flag. gufo picks its own KV dtype; there is no --cache-type-k. The old CACHE_RAM=8192 tunable is gone with it. Prompt reuse is on by default, and --cache-disk (restart-safe) is opt-in and currently off.

• Health/readiness split: GET /health is process liveness and is what the deployer gates on; GET /ready is model/backend readiness. GET /metrics is Prometheus-format.

• Long bodies: --max-request-bytes defaults to 8 MiB.

Failure modes worth knowing

• A missing runtime is a deploy error, not a restart loop. The handler gates on LLM_LARGE_RUNTIME before it stops anything, and the serve script exits with a build-gufo.sh pointer rather than exec'ing.
• Host memory is the ceiling, and it is shared. frank-large and ComfyUI both hold gfx1151 compute queues. gufo reads /sys/class/kfd/kfd/proc/*/queues/*/type and caps itself (GPU_MAX_HW_QUEUES, 2 resident for serve llm) so a full device time-slices instead of failing; four HIP processes is the device ceiling.
• Load is seconds, not minutes (9.2 s measured cold on the Swift target), so a redeploy no longer spends 20 minutes in LLM_WAIT.
• A payload is two files in two directories. The target and the drafter no longer live together, so "the payload is present" is not one check; the deployer gates on both paths and the serve script refuses to start if either is absent. A symlinked "one payload directory" would hide that.
• A v0.1.x runtime is a moving target. Rebuilds are not automatic — RUNTIME_DIR names a version, and the deployer will not silently substitute another.

Rollback

Two independent rollbacks, because two things changed on 2026-09-28:

1. Payload only — repoint MODEL_FILE/MODEL_DIR in scripts/frank-large.env at /home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf and redeploy. That directory is untouched on disk and the drafter already lives in it. This is the fast way back to the stock Qwen weights on the gufo runtime.
2. Engine — frank-large-unsloth-qwen38-27b-q8xl: the unsloth b11030 prebuilt is still at /home/dank/GitPackages/unsloth-llama-b11030/. Revert RUNTIME_DIR and the gufo-only tunables, restore the serve script's llama.cpp command, and redeploy. This is also the only way to get qwen-sharp-v22.4.0.jinja back.

Citations

• Runtime: https://github.com/gufo-org/gufo (pinned v0.1.1, MIT)
• Measured tables: docs/models/qwen3.8-27b/BENCHMARKS.md in that repo
• API contract: docs/SERVER.md; flags: docs/CLI.md; the C++ renderer that rules out a Jinja template: src/models/qwen/chat_template.cpp (587 lines)
• GGUF header evidence for the payload swap: both files' headers were parsed and diffed directly (architecture, all 866 tensor names in order, per-tensor ggml_type and dims) before either was loaded
• Build: scripts/build-gufo.sh; serve: scripts/serve-frank-large.sh; tunables: scripts/frank-large.env
• Payload: scripts/download-frank-large-swift.sh (target), scripts/download-frank-large.sh (DFlash2 drafter)
• Predecessor: frank-large-unsloth-qwen38-27b-q8xl