Frank
aaef419a6d
gufo is a vertical inference engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large has swapped runtimes seven times in search of a better one. This one ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on the host and the payload, port, alias and client contract stay exactly where they are. Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1, built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4 through its RPATH — the opposite of the unsloth prebuilt it replaces, which carried its own — so two host workarounds are encoded in that script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set that shadows AMD's and cannot be overridden with -I in HIP mode, and the ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose libstdc++ is actually installed. Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL target and the same DFlash2 drafter, greedy, text-only. Speculation moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy capped at 7 proposals. Client-contract deltas, all documented: the chat template is compiled into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token failure mode is gone), reasoning_effort now takes the full OpenAI set (so models.yml's reasoningEffortMap fold is no longer required — kept anyway, so this commit does not also move every high role from xhigh to high effort), and there is no KV-quant or prompt-cache-RAM flag. The test seam now runs the serve script against a stub engine and asserts on the argv it actually execs, instead of grepping the script's text. The deploy handler gates on the runtime binary and returns before it stops the serving unit. Measured trade, from gufo's own Q8_K_XL tables against llama.cpp b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the 8-bit target the crossover is ~32k, and frank-large runs one 262,144 conversation. The Q4_K_XL target shows no such regression.
118 lines
5.3 KiB
Markdown
118 lines
5.3 KiB
Markdown
---
|
|
type: reference
|
|
title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
|
description: "The RETIRED frank-large runbook (2026-09-21 → 2026-09-28; replaced by [[frank-large-gufo-qwen38-27b]], and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
|
|
tags:
|
|
- llm
|
|
- llama.cpp
|
|
- unsloth
|
|
- dankclaw
|
|
- frank-large
|
|
- strix-halo
|
|
- gfx1151
|
|
- q8
|
|
- dflash2
|
|
- rocm
|
|
- q8-kv
|
|
timestamp: 2026-09-21
|
|
---
|
|
|
|
# frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
|
|
|
|
> **RETIRED 2026-09-28**, replaced by [[frank-large-gufo-qwen38-27b]] (the
|
|
> gufo engine, same weights, same port and alias); the prebuilt is intact on
|
|
> disk as the rollback target. ACTIVE from 2026-09-21, when it replaced
|
|
> [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge IU4 W4A4) and
|
|
> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it
|
|
> preceded).
|
|
|
|
## Identity
|
|
|
|
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
|
- **Runtime:** stock llama.cpp — the unsloth `b11030` prebuilt
|
|
(`/home/dank/GitPackages/unsloth-llama-b11030/llama-server`), ROCm gfx1151.
|
|
A mainline build, not a fork: it supports the `qwen3_5` architecture and the
|
|
DFlash2 drafter (`llama_model_qwen35`, `llama_model_dflash`,
|
|
`common_speculative_impl_draft_dflash`).
|
|
- **Service shape:** root systemd unit `scripts/frank-large.service`
|
|
(`User=dank`), launched by `scripts/serve-frank-large.sh` with tunables from
|
|
`scripts/frank-large.env`.
|
|
|
|
## Shape
|
|
|
|
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a 27B **dense** model,
|
|
31.46 GiB GGUF (sha256-verified by `scripts/download-frank-large.sh`).
|
|
- **Context:** one stream, full 262,144 tokens (the model's native window),
|
|
`PARALLEL_SLOTS=1`.
|
|
- **Placement:** all layers on the iGPU (`--n-gpu-layers 999`). The 27B dense
|
|
weights (31.46 GiB) + the DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) +
|
|
compute buffers fit the iGPU's GTT aperture (~114 GiB).
|
|
|
|
## Spec method
|
|
|
|
DFlash2 (a block-diffusion drafter) at **depth 4**. The drafter is the z-lab
|
|
`Qwen3.8-27B-DFlash2-Q4_K_M` (1.14 GiB), loaded via `--model-draft` with
|
|
`--spec-type draft-dflash` and `--spec-draft-device ROCm0`. It has a sliding
|
|
window of 2048, so its draft KV is small (~40 MiB). Greedy verification
|
|
(`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) is required.
|
|
|
|
## KV
|
|
|
|
**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), ~8.6 GiB at 262k. The
|
|
unsloth b11030 build does **not** support fp8 KV (allowed: f32/f16/bf16/q8_0/
|
|
q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the highest-precision 8-bit option.
|
|
|
|
## Template
|
|
|
|
The Qwen-Sharp v22.4.0 template is pinned via `--chat-template-file`
|
|
(`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja`) with `--jinja` and
|
|
`--chat-template-kwargs '{"preserve_thinking":true}'`. Without
|
|
`--chat-template-file` the binary falls back to the built-in `peg-native`
|
|
template, which produces a ~73,600-token prompt for a ~250-token request.
|
|
`preserve_thinking` keeps prior thinking blocks across turns for multi-turn
|
|
agents. The template gates `reasoning_effort` to `{xhigh, medium, low}` — the
|
|
harness folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
|
|
|
|
## Payload
|
|
|
|
`scripts/download-frank-large.sh` fetches the main GGUF and the DFlash2
|
|
drafter, each verified against its declared byte count + sha256
|
|
(`--verify-only` re-checks). The serve script refuses to start if the binary,
|
|
the main GGUF, or the drafter is missing.
|
|
|
|
## Client contract
|
|
|
|
- **Alias:** `frank-large` (the `id` at `/v1/models`).
|
|
- **Reasoning:** extracted to `reasoning_content` (`--reasoning auto`,
|
|
`--reasoning-format deepseek`, unlimited budget).
|
|
- **Tools:** supported (`--jinja` is pinned).
|
|
- **Vision:** none — text-only (no mmproj), matching the prior frank-large.
|
|
- **Sampling:** greedy by default; a client that sends a temperature gets its
|
|
own, and the harness applies per-model overrides from `models.yml`.
|
|
|
|
## Failure modes worth knowing
|
|
|
|
- **Model load is slow (minutes cold).** Health returns 503 while loading; do
|
|
not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`).
|
|
- **`--ngl` (double-dash) is invalid in b11030** — use `--n-gpu-layers`
|
|
(the serve script uses the long form).
|
|
- **The 8-bit UD-Q8_K_XL decodes ~17 t/s** vs the prior 4-bit Kairic Edge IU4
|
|
at ~22 t/s — the ~23 % larger weight size (31.46 vs 26 GiB) explains the
|
|
gap; it is not a regression.
|
|
- **DFlash2 draft acceptance is text-variety-dependent** (0.85 on a trivial
|
|
one-word answer, 0.519 on a varied explanatory prompt) — do not read a
|
|
single-request acceptance as the steady state.
|
|
|
|
## Rollback
|
|
|
|
Payload and runtime for [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge
|
|
IU4 W4A4) remain intact on disk. See [[frank-large-unsloth-qwen38-flash-next]]
|
|
for the Flash-Next predecessor it replaced on 2026-09-21.
|
|
|
|
# Citations
|
|
|
|
- unsloth prebuilts: https://huggingface.co/unsloth
|
|
- Model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
|
|
- DFlash2 drafter: https://huggingface.co/z-lab
|
|
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
|