Frank
9d9d28cda6
Live numbers from the 2026-09-28 deploy: 13 s load to listening, RSS 30.6 GiB, ttft 549.8 ms / inter-token 24.2 ms on a short prompt, and a green contract battery (health, ready, /v1/models at 262,144, content + reasoning_content, tool_calls, SSE, and a two-turn tool loop). Also recorded: gufo accepts every request field the harness sends in extraBody and the full reasoning_effort set, both verified 200 — which is why models.yml's reasoningEffortMap fold is now optional rather than required, and why it is being kept anyway.
195 lines
9.7 KiB
Markdown
195 lines
9.7 KiB
Markdown
---
|
||
type: reference
|
||
title: "gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||
description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the one-session / full-window shape, what changed in the client contract versus llama.cpp, the measured prefill/decode trade, and the failure modes worth knowing."
|
||
tags:
|
||
- llm
|
||
- gufo
|
||
- unsloth
|
||
- dankclaw
|
||
- frank-large
|
||
- strix-halo
|
||
- gfx1151
|
||
- q8
|
||
- dflash2
|
||
- rocm
|
||
timestamp: 2026-09-28
|
||
---
|
||
|
||
# frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2)
|
||
|
||
> **ACTIVE since 2026-09-28.** Replaces
|
||
> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
|
||
> b11030 prebuilt), which had replaced
|
||
> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
|
||
> alias (`frank-large`), **same weights**.
|
||
|
||
## Identity
|
||
|
||
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
||
- **Runtime:** [gufo](https://github.com/gufo-org/gufo) — "the Strix Halo
|
||
inference engine", a vertical engine built for exactly this hardware
|
||
(Ryzen AI MAX+ 395 / Radeon 8060S, `gfx1151`, unified memory). Pinned at
|
||
**v0.1.1** (commit `b0f8673`), built from source on the host to
|
||
`/home/dank/GitPackages/gufo-v0.1.1/`.
|
||
- **Service shape:** root systemd unit `scripts/frank-large.service`
|
||
(`User=dank`, `LimitMEMLOCK=infinity`), launched by
|
||
`scripts/serve-frank-large.sh` with tunables from
|
||
`scripts/frank-large.env`.
|
||
|
||
gufo ships **no prebuilt Linux binaries** (the v0.1.0/v0.1.1 release assets
|
||
are empty) and no Nix on this host, so the runtime is a source build. It is
|
||
also young: v0.1.0 landed 2026-09-28, the same day as this cutover.
|
||
|
||
## Why this engine
|
||
|
||
gufo's own measurement of the two engines on the same Qwen3.8-27B targets
|
||
(Strix Halo, `628ed18e` vs llama.cpp `b11069`, greedy, DFlash2):
|
||
|
||
| Q8_K_XL, DFlash2 | gufo | llama.cpp | delta |
|
||
|---|---:|---:|---:|
|
||
| prefill, depth 0 | 457.05 t/s | 333.85 t/s | **+36.9 %** |
|
||
| prefill, depth 131,072 | 237.24 t/s | 131.80 t/s | **+80.0 %** |
|
||
| decode, depth 0 (mixed) | 16.08 t/s | 15.00 t/s | +7.2 % |
|
||
| decode, depth 32,768 (mixed) | 14.41 t/s | 14.44 t/s | −0.2 % |
|
||
| decode, depth 65,536 (mixed) | 10.87 t/s | 12.62 t/s | **−13.9 %** |
|
||
| decode, depth 131,072 (mixed) | 8.87 t/s | 11.62 t/s | **−23.7 %** |
|
||
| cold load to HTTP-ready | 8.17 s | 36.47 s | **4.5× faster** |
|
||
| peak memory @ 262,144 ctx | 50.58 GiB | 48.60 GiB | +4 % |
|
||
|
||
Read that honestly: **prefill, load time and short-context decode improve;
|
||
deep-context decode on the 8-bit target gets worse.** The crossover on Q8 is
|
||
around 32k of cached depth. frank-large runs one 262,144-token conversation,
|
||
so a long agent session spends its life on the losing side of that line
|
||
while a short turn wins. The Q4_K_XL target does not show the regression
|
||
(gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to
|
||
revisit the quantization separately from the runtime.
|
||
|
||
## Build
|
||
|
||
`scripts/build-gufo.sh` reproduces the runtime end to end. It clones the
|
||
pinned tag, configures the `release` preset (HIP, `--offload-arch=gfx1151`),
|
||
builds, installs, and finishes with `gufo diagnose`. Two host facts it
|
||
encodes, both found the hard way on 2026-09-28:
|
||
|
||
- **Ubuntu's `librocwmma-dev` must not be installed.** It ships a truncated
|
||
rocWMMA header set (the public headers without `rocwmma/internal/`) into
|
||
`/usr/include`, where it shadows AMD's complete copy under
|
||
`/opt/rocm/include`. The first build died on `'internal/accessors.hpp'
|
||
file not found` from `/usr/include/rocwmma/rocwmma.hpp`, and **a `-I`
|
||
cannot fix it**: in HIP mode the host compilation pass searches the system
|
||
directories first. The package has no reverse dependencies; the script
|
||
removes it.
|
||
- **`--gcc-install-dir` must point at GCC 15.** Ubuntu 26.04 has
|
||
`/usr/include/c++/16`, and the ROCm clang picks the newest tree, whose
|
||
`<atomic_wait.h>` references `std::__detail::__notify_impl` at a symbol
|
||
version the installed `libstdc++.so.6` (GLIBCXX_3.4.35) does not export.
|
||
The link then fails with `undefined symbol`. Pinning the headers to the
|
||
GCC the runtime library came from fixes it.
|
||
|
||
Verify an installed runtime with `bash scripts/build-gufo.sh --verify`, or
|
||
`gufo diagnose` directly — it reports `[PASS]` plus the detected
|
||
architecture, driver and queue budget.
|
||
|
||
## Shape
|
||
|
||
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — 27B dense,
|
||
29.30 GiB. **Unchanged by the cutover**, still fetched and sha256-verified
|
||
by `scripts/download-frank-large.sh`.
|
||
- **Context:** one session, full 262,144 tokens (the model's native window),
|
||
`SESSIONS=1`. The fleet's single stream is unchanged.
|
||
- **Spec method:** DFlash2, gufo's **adaptive** block policy
|
||
(`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the same
|
||
z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB).
|
||
- **Sampling:** greedy (`--temperature 0`), as every prior frank-large
|
||
served. gufo's per-model recommendation for Qwen3.8 thinking is *sampled*
|
||
(1.0 / 0.95 / top-k 20) — that is a separate decision, deliberately not
|
||
bundled into a runtime swap.
|
||
- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
|
||
with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
|
||
|
||
### Measured at the 2026-09-28 cutover
|
||
|
||
Live on Dankclaw after `deploy.sh dankclaw --service frank-large`:
|
||
|
||
- **13 s** from `load_started` to `listening` (target 8 s, drafter 2 s,
|
||
session 3 s), against a 20-minute `LLM_WAIT` budget — a redeploy no longer
|
||
sits in a load wait.
|
||
- **RSS 30.6 GiB**, VSZ 87.3 GiB; host `used` 81 GiB of 121 GiB, i.e. the
|
||
same order as the llama.cpp deployment it replaced.
|
||
- A short-prompt live request returned
|
||
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2, max_inter_token;dur=166.9`
|
||
(≈41 tok/s). Short-prompt decode is gufo's *winning* regime; see the depth
|
||
table above before quoting it as the endpoint's number.
|
||
- Contract battery, all green: `/health` 200, `/ready` 200,
|
||
`/v1/models` → `frank-large` with `context_length 262144`, a chat
|
||
completion carrying both `content` and `reasoning_content`, a `tool_calls`
|
||
response with `finish_reason: tool_calls`, and SSE streaming.
|
||
|
||
## Client contract — what actually changed
|
||
|
||
Everything the fleet depended on still holds: port 7770, alias `frank-large`,
|
||
`reasoning_content`, `tool_calls`, streaming SSE, and text-only (gufo
|
||
auto-discovers a `mmproj` beside the model; this payload has none).
|
||
|
||
The differences worth knowing:
|
||
|
||
- **No custom chat template.** gufo compiles the official Qwen Jinja into the
|
||
binary and takes no `--chat-template-file`. The Qwen-Sharp v22.4.0 file
|
||
this endpoint used with llama.cpp is **no longer in the command line** —
|
||
the 2026-09-21 "73,600-token prompt" incident cannot recur, because there
|
||
is no fallback template to land in. Thinking defaults to on at `xhigh`
|
||
with prior reasoning preserved (`--think auto --preserve-thinking on`,
|
||
spelled out in the serve script).
|
||
- **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`,
|
||
`low`, `medium`, `high`, `xhigh`, `max`). Qwen-Sharp rejected anything
|
||
outside `{xhigh, medium, low}` with an HTTP 500, which is why `models.yml`
|
||
still carries a `reasoningEffortMap` fold for this model. **The fold is
|
||
kept on purpose**: removing it would silently move every `high` role from
|
||
`xhigh` to `high` effort in the same commit as the runtime swap. Drop it as
|
||
its own change once the effort curve is a decision rather than a side
|
||
effect.
|
||
- **No KV-cache quantization flag.** gufo picks its own KV dtype; there is
|
||
no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it.
|
||
Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in
|
||
and currently off.
|
||
- **Health/readiness split:** `GET /health` is process liveness and is what
|
||
the deployer gates on; `GET /ready` is model/backend readiness.
|
||
`GET /metrics` is Prometheus-format.
|
||
- **Long bodies:** `--max-request-bytes` defaults to 8 MiB and unknown
|
||
request fields are rejected rather than ignored.
|
||
|
||
## Failure modes worth knowing
|
||
|
||
- **A missing runtime is a deploy error, not a restart loop.** The handler
|
||
gates on `LLM_LARGE_RUNTIME` *before* it stops anything, and the serve
|
||
script exits with a `build-gufo.sh` pointer rather than exec'ing.
|
||
- **Host memory is the ceiling, and it is shared.** frank-large and ComfyUI
|
||
both hold gfx1151 compute queues. gufo reads
|
||
`/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself
|
||
(`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device
|
||
time-slices instead of failing; four HIP processes is the device ceiling.
|
||
- **Load is seconds, not minutes** (8 s measured cold), so a redeploy no
|
||
longer spends 20 minutes in `LLM_WAIT`.
|
||
- **A v0.1.x runtime is a moving target.** Rebuilds are not automatic —
|
||
`RUNTIME_DIR` names a version, and the deployer will not silently
|
||
substitute another.
|
||
|
||
## Rollback
|
||
|
||
[[frank-large-unsloth-qwen38-27b-q8xl]] — the unsloth b11030 prebuilt is
|
||
still at `/home/dank/GitPackages/unsloth-llama-b11030/` and the payload is
|
||
the same directory the current runtime reads. Roll back by reverting
|
||
`RUNTIME_DIR` (and the gufo-only tunables) in `scripts/frank-large.env`,
|
||
restoring the serve script's llama.cpp command, and redeploying.
|
||
|
||
# Citations
|
||
|
||
- Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT)
|
||
- Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo
|
||
- API contract: `docs/SERVER.md`; flags: `docs/CLI.md`
|
||
- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`;
|
||
tunables: `scripts/frank-large.env`
|
||
- Payload: `scripts/download-frank-large.sh`
|
||
- Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]]
|