serverAppConfigs/serverAppConfigs-docs/LLM/frank-dragoon-gufo-qwen38-27b.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-dragoon)	The RETIRED frank-dragoon runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing.	llm gufo swift-1.5 dankclaw frank-dragoon strix-halo gfx1151 q8 dflash2 rocm	2026-09-28

frank-dragoon — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)

RETIRED 2026-10-02 — the rollback target. Superseded by frank-dragoon-rocmfpx-q8agent-mtp (Q8_0_ROCMFPX_AGENT at 8.39 bpw with MTP4, 17.0 t/s measured). This era was itself a return after one day of frank-dragoon-vllm-mimo-v26-bf16 (battery decode 6.19-9.01 t/s; every ≥6-bit vLLM quant path dead on gfx1151). The shape moved on the way back: payload is now unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (16.35 GiB, vanilla — the abliterated Swift Q8 was reclaimed), and the session grid is one session at the full native 262,144 window (SESSIONS=1 CONTEXT_SIZE=262144, f16 KV). The Swift-1.5 Q8 history below is retained for the engine/template law, which still governs. Originally ACTIVE 2026-09-28. Replaces frank-dragoon-unsloth-qwen38-27b-q8xl (stock llama.cpp, the unsloth b11030 prebuilt), which had replaced frank-dragoon-kairic-edge-qwen38-27b. Same host, same port (7770), same alias (frank-dragoon).

The payload also changed on 2026-09-28, later the same day: the target went from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL to ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's UD-Q8_K_XL — an abliterated (uncensored) fine-tune of the same base. See Payload and, for what that costs, the template law.

Identity

• Endpoint: frank-dragoon at 192.168.0.143:7770 (Dankclaw).
• Runtime: gufo — "the Strix Halo inference engine", a vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151, unified memory). Pinned at v0.1.1 (commit b0f8673), built from source on the host to /home/dank/GitPackages/gufo-v0.1.1/.
• Service shape: root systemd unit scripts/frank-dragoon.service (User=dank, LimitMEMLOCK=infinity), launched by scripts/serve-frank-dragoon.sh with tunables from scripts/frank-dragoon.env.

gufo ships no prebuilt Linux binaries (the v0.1.0/v0.1.1 release assets are empty) and no Nix on this host, so the runtime is a source build. It is also young: v0.1.0 landed 2026-09-28, the same day as this cutover.

Why this engine

gufo's own measurement of the two engines on the same Qwen3.8-27B targets (Strix Halo, 628ed18e vs llama.cpp b11069, greedy, DFlash2):

Q8_K_XL, DFlash2	gufo	llama.cpp	delta
prefill, depth 0	457.05 t/s	333.85 t/s	+36.9 %
prefill, depth 131,072	237.24 t/s	131.80 t/s	+80.0 %
decode, depth 0 (mixed)	16.08 t/s	15.00 t/s	+7.2 %
decode, depth 32,768 (mixed)	14.41 t/s	14.44 t/s	−0.2 %
decode, depth 65,536 (mixed)	10.87 t/s	12.62 t/s	−13.9 %
decode, depth 131,072 (mixed)	8.87 t/s	11.62 t/s	−23.7 %
cold load to HTTP-ready	8.17 s	36.47 s	4.5× faster
peak memory @ 262,144 ctx	50.58 GiB	48.60 GiB	+4 %

Read that honestly: prefill, load time and short-context decode improve; deep-context decode on the 8-bit target gets worse. The crossover on Q8 is around 32k of cached depth. frank-dragoon runs one 262,144-token conversation, so a long agent session spends its life on the losing side of that line while a short turn wins. The Q4_K_XL target does not show the regression (gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to revisit the quantization separately from the runtime.

Build

scripts/build-gufo.sh reproduces the runtime end to end. It clones the pinned tag, configures the release preset (HIP, --offload-arch=gfx1151), builds, installs, and finishes with gufo diagnose. Two host facts it encodes, both found the hard way on 2026-09-28:

• Ubuntu's librocwmma-dev must not be installed. It ships a truncated rocWMMA header set (the public headers without rocwmma/internal/) into /usr/include, where it shadows AMD's complete copy under /opt/rocm/include. The first build died on 'internal/accessors.hpp' file not found from /usr/include/rocwmma/rocwmma.hpp, and a -I cannot fix it: in HIP mode the host compilation pass searches the system directories first. The package has no reverse dependencies; the script removes it.
• --gcc-install-dir must point at GCC 15. Ubuntu 26.04 has /usr/include/c++/16, and the ROCm clang picks the newest tree, whose <atomic_wait.h> references std::__detail::__notify_impl at a symbol version the installed libstdc++.so.6 (GLIBCXX_3.4.35) does not export. The link then fails with undefined symbol. Pinning the headers to the GCC the runtime library came from fixes it.

Verify an installed runtime with bash scripts/build-gufo.sh --verify, or gufo diagnose directly — it reports [PASS] plus the detected architecture, driver and queue budget.

Shape

Payload

• Model: ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF, file UD-Q8_K_XL — 27B dense, 31,457,990,272 bytes, fetched and sha256-verified (9076f99a…, the Hub LFS oid) by scripts/download-frank-dragoon-swift.sh. It is an abliterated (uncensored) fine-tune of Qwen3.8-27B. The repository also publishes an mmproj-BF16.gguf, which is not used: this endpoint is text-only.
• Licence: the repo declares other — the SWIFT OPEN LICENSE 1.0, which is not an OSI licence. That is a real constraint on redistribution, noted here so nobody discovers it later. It is fine for a private fleet.
• Replaced: unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL, still on disk and still the rollback target for the payload.

The two targets are structurally interchangeable for this engine, which is why the swap needed no engine change. Header-diffed directly:

	unsloth	Swift-1.5
general.architecture	qwen35	qwen35
tensors (count, names, order)	866	866, identical
per-tensor ggml_type	Q8_0×453, F32×360, BF16×53	identical
per-tensor dims	—	0 mismatches
MTP head	4 tensors at blk.64.nextn.*	identical
geometry	65 blk, 5120 embd, 17408 FFN, 24/4 heads	identical

The two metadata differences that look alarming are both no-ops, and both were confirmed against gufo's source rather than assumed:

• general.file_type is 15 vs 7. gufo's gguf_reader.cpp states it is "deliberately not consulted… The Unsloth UD-*_K_XL artifacts this repo targets carry a stale one — UD-Q8_K_XL records ftype 15 (Q4_K_M) … The tensor table always is." The file gufo already served lies about itself here, so gufo reads the tensor table.
• Swift adds qwen35.attention.recurrent_layers. gufo never reads that key anywhere in src/; layer kinds come from full_attention_interval, which both files carry as 4 and which agrees with it.

Serving shape

• Context: one session, full 262,144 tokens (the model's native window), SESSIONS=1. The fleet's single stream is unchanged.
• Spec method: DFlash2, gufo's adaptive block policy (--draft-policy adaptive, --draft-tokens 7). The drafter is the unchanged z-lab Qwen3.8-27B-DFlash2-Q4_K_M (1.07 GiB) — it did not move with the target, which is why DRAFTER_DIR is a separate tunable from MODEL_DIR and why the deployer's stray-run sweep keys on the drafter's filename rather than the target's.
• Sampling: greedy (--temperature 0) server-side, as every prior frank-dragoon served. The harness overrides per request via extraBody (1.0 / 0.95 / top-k 20).
• Memory: gufo's own table puts the 8-bit target at 50.58 GiB peak with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.

Measured at the 2026-09-28 cutover

Live on Dankclaw. The target numbers are the Swift-1.5 load test (a standalone run under the production flags, service stopped first); the engine numbers are from the gufo cutover earlier the same day and are unchanged by the payload swap.

• 9.2 s load_started → load_completed for the Swift target, against a 20-minute LLM_WAIT budget. The unsloth target loaded in 13 s.
• 66.5 GiB of the 116.7 GiB iGPU device at context capacity 262,144; host RSS 30.6 GiB. No loader warnings, no skipped sections.
• Contract battery, all green on the Swift target: /v1/models → context_length 262144, a chat completion carrying both content and reasoning_content, a tool_calls response with finish_reason: tool_calls, SSE streaming, and a two-turn tool loop (tool call → tool result → follow-up) with prior reasoning preserved. Arithmetic spot-check: "17*23" → 391.
• Behavioural spot-check that this is the abliterated model: asked for a two-sentence lockpicking scene, it wrote one without hedging or refusing.
• From the gufo cutover, on the unsloth target: a short-prompt live request returned Server-Timing: ttft;dur=549.8, inter_token;dur=24.2 (≈41 tok/s). Short-prompt decode is gufo's winning regime; see the depth table above before quoting it as the endpoint's number.

Tuning: the served flags are already optimal (2026-09-28)

Frank asked for iterative decode/prefill improvement at a 32k working point without touching the model or the quantization. Result: no win exists at this engine's served settings, and nothing was changed. Measured on real repo prose and shell sources at 32,762 prompt tokens, salted per run so a warm prefix cache could not help, 3–5 repetitions each.

lever	range swept	outcome
--prefill-chunk	512 → 8192	no effect — 410.7–412.4 t/s prefill across a 16× change
--draft-tokens	5 → 10	flat — 13.79–15.86 t/s decode, inside run-to-run noise
--draft-policy	fixed vs adaptive	35.90 vs 35.50 (bench) — no effect
--min-draft-tokens	—	rejected by the engine: "DFlash2 requires … 1"
MTP backend	—	unsupported for Q8: token_embd.weight is Q8_0
--context 262144 → 40960	diagnostic	13.93 vs 14.71 t/s — no gain, and it would break the declared client context

The confirmation round at 5 reps reversed the 3-rep leaders: the pair that looked best (dt9 + chunk4096) came last at 12.86 t/s, and the deployed configuration came first at 14.71 t/s. Every difference sits inside the noise band.

What does move the number is speculative decoding itself: 6.65 t/s off → ~15 t/s on, a 5.3× step — and that is already switched on. Prefill is insensitive to everything tried, which is consistent with gufo's own EXPERIMENTS.md, where dozens of kernel-level alternatives are recorded as rejected for "no release throughput improvement".

Do not repeat this investigation without new information. Two traps cost most of the time, and both are in AGENTS.md's benchmarking policy: gufo bench reports acceptance=0.933 ± 0.000 on an identical trace hash and inverts the draft-width ranking; and a live A/B needs ≥128 generated tokens or inter_token is first-token noise.

The template law: Qwen-Sharp is gone

scripts/qwen-sharp-v22.4.0.jinja is the fleet's chat template — it is wired into frank-archon as R9V_CHAT_TEMPLATE_FILE and it was frank-dragoon's --chat-template-file under llama.cpp. frank-dragoon can no longer use it, and the reason is stronger than "gufo lacks the flag":

gufo has no Jinja engine at all. src/models/qwen/chat_template.cpp is 587 lines of hand-written C++ that appends strings to build ChatML. There is no --chat-template-file, no environment override, and no template body anywhere in the binary to point at. Qwen-Sharp is 453 lines of Jinja2 using namespace(), macros, raise_exception and image/video branches — supplying it would mean reimplementing it in C++, not handing gufo a file.

Given the choice on 2026-09-28, the decision was to ship on gufo's stock Jinja and accept the loss. What that gives up, concretely:

Qwen-Sharp feature	On gufo
terse system-prompt injection (the headline behaviour)	gone
in-band <|think_off|> / <|think_xhigh|> control	gone
tool_call_format json | xml switch	gone
max_tool_arg_chars / max_tool_response_chars truncation	gone
add_vision_id → Picture N: prefixes	gone
effort instruction strings in the system prompt	kept (gufo has its own, same text)
preserve_thinking	kept (--preserve-thinking on)

The terse injection is the one that matters most here, and it is worth being blunt about the combination: this endpoint now runs an abliterated model without the prompt that tells it to stop waffling. If verbosity shows up, the cheapest fix is to send the terse text as an ordinary system message from the client (models.yml / the harness) rather than through a template — that recovers the behaviour without forking the renderer. The full-fidelity options are reverting to llama.cpp or porting Qwen-Sharp into the C++ renderer; both are worse trades today, and the second is the same class of change that produced the 2026-09-21 "73,600-token prompt" incident.

Client contract — what actually changed

Everything the fleet depended on still holds: port 7770, alias frank-dragoon, reasoning_content, tool_calls, streaming SSE, and text-only (gufo auto-discovers a mmproj beside the model; this payload has none).

The differences worth knowing:

• No custom chat template — see The template law above, which is the most consequential change on this endpoint. Thinking defaults to on at xhigh with prior reasoning preserved (--think auto --preserve-thinking on, spelled out in the serve script).

• reasoning_effort accepts the full OpenAI set (none, minimal, low, medium, high, xhigh, max), all verified 200 against the live endpoint. Qwen-Sharp rejected anything outside {xhigh, medium, low} with an HTTP 500, which is why models.yml still carries a reasoningEffortMap fold for this model. The fold is kept on purpose: removing it would silently move every high role from xhigh to high effort. Drop it as its own change.

• Every field the harness sends is accepted. top_k, min_p, presence_penalty, repetition_penalty and preserve_thinking are all outside gufo's documented request subset, and all returned 200. They are applied from models.yml's extraBody unchanged.

• No KV-cache quantization flag. gufo picks its own KV dtype; there is no --cache-type-k. The old CACHE_RAM=8192 tunable is gone with it. Prompt reuse is on by default, and --cache-disk (restart-safe) is opt-in and currently off.

• Health/readiness split: GET /health is process liveness and is what the deployer gates on; GET /ready is model/backend readiness. GET /metrics is Prometheus-format.

• Long bodies: --max-request-bytes defaults to 8 MiB.

Failure modes worth knowing

• A missing runtime is a deploy error, not a restart loop. The handler gates on LLM_DRAGOON_RUNTIME before it stops anything, and the serve script exits with a build-gufo.sh pointer rather than exec'ing.
• Host memory is the ceiling, and it is shared. frank-dragoon and ComfyUI both hold gfx1151 compute queues. gufo reads /sys/class/kfd/kfd/proc/*/queues/*/type and caps itself (GPU_MAX_HW_QUEUES, 2 resident for serve llm) so a full device time-slices instead of failing; four HIP processes is the device ceiling.
• Load is seconds, not minutes (9.2 s measured cold on the Swift target), so a redeploy no longer spends 20 minutes in LLM_WAIT.
• A payload is two files in two directories. The target and the drafter no longer live together, so "the payload is present" is not one check; the deployer gates on both paths and the serve script refuses to start if either is absent. A symlinked "one payload directory" would hide that.
• A v0.1.x runtime is a moving target. Rebuilds are not automatic — RUNTIME_DIR names a version, and the deployer will not silently substitute another.

Rollback

Two independent rollbacks, because two things changed on 2026-09-28:

1. Payload only — repoint MODEL_FILE in scripts/frank-dragoon.env at /home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf and redeploy. The 29.3 GiB target was reclaimed on 2026-09-28, so this path now needs a re-fetch first: scripts/download-frank-dragoon.sh, then the env edit. The DFlash2 drafter was never touched and still lives in that directory.
2. Engine — frank-dragoon-unsloth-qwen38-27b-q8xl: the unsloth b11030 prebuilt is still at /home/dank/GitPackages/unsloth-llama-b11030/. Revert RUNTIME_DIR and the gufo-only tunables, restore the serve script's llama.cpp command, and redeploy. This is also the only way to get qwen-sharp-v22.4.0.jinja back.

Citations

• Runtime: https://github.com/gufo-org/gufo (pinned v0.1.1, MIT)
• Measured tables: docs/models/qwen3.8-27b/BENCHMARKS.md in that repo
• API contract: docs/SERVER.md; flags: docs/CLI.md; the C++ renderer that rules out a Jinja template: src/models/qwen/chat_template.cpp (587 lines)
• GGUF header evidence for the payload swap: both files' headers were parsed and diffed directly (architecture, all 866 tensor names in order, per-tensor ggml_type and dims) before either was loaded
• Build: scripts/build-gufo.sh; serve: scripts/serve-frank-dragoon.sh; tunables: scripts/frank-dragoon.env
• Payload: scripts/download-frank-dragoon-swift.sh (target), scripts/download-frank-dragoon.sh (DFlash2 drafter)
• Predecessor: frank-dragoon-unsloth-qwen38-27b-q8xl