dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: replace stock llama.cpp with gufo (gufo-org/gufo) on Dankclaw

Browse Source 
gufo is a vertical inference engine built for exactly this hardware
(Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large
has swapped runtimes seven times in search of a better one. This one
ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on
the host and the payload, port, alias and client contract stay exactly
where they are.

Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1,
built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4
through its RPATH — the opposite of the unsloth prebuilt it replaces,
which carried its own — so two host workarounds are encoded in that
script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set
that shadows AMD's and cannot be overridden with -I in HIP mode, and the
ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose
libstdc++ is actually installed.

Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL
target and the same DFlash2 drafter, greedy, text-only. Speculation
moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy
capped at 7 proposals.

Client-contract deltas, all documented: the chat template is compiled
into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token
failure mode is gone), reasoning_effort now takes the full OpenAI set
(so models.yml's reasoningEffortMap fold is no longer required — kept
anyway, so this commit does not also move every high role from xhigh to
high effort), and there is no KV-quant or prompt-cache-RAM flag.

The test seam now runs the serve script against a stub engine and
asserts on the argv it actually execs, instead of grepping the script's
text. The deploy handler gates on the runtime binary and returns before
it stops the serving unit.

Measured trade, from gufo's own Q8_K_XL tables against llama.cpp
b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero
depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the
8-bit target the crossover is ~32k, and frank-large runs one 262,144
conversation. The Q4_K_XL target shows no such regression.
This commit is contained in:
Frank
2026-09-28 22:08:29 -07:00
parent d29d104f01
commit aaef419a6d
13 changed files with 695 additions and 240 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+70 -37
View File
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 |
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 29.3 GiB) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, compiled-in Qwen Jinja, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in
@@ -227,7 +227,7 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
merely wedged kernel usually still answers ARP.
1. Stop and mask (self-healing would restart it):
- Dankclaw `frank-large` (root unit, stock llama.cpp on the iGPU):
- Dankclaw `frank-large` (root unit, gufo on the iGPU):
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
@@ -277,47 +277,79 @@ FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
projector's audio encoder (frank-small transcribes WAV verbatim even though
`audio` is omitted from its `input` list — OMP's schema rejects it).
### frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
### frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
The unsloth llama.cpp prebuilt `app-b11030-mix-5ff778e` (ROCm gfx1151,
`/home/dank/GitPackages/unsloth-llama-b11030/`, bundles its own ROCm
runtime — no host SDK gate) as a **root systemd unit**
**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a
vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 /
Radeon 8060S, `gfx1151`) — as a **root systemd unit**
(`scripts/frank-large.service`, `User=dank`), serving
`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 31.5 GiB,
sha256-verified) on port 7770 from the gfx1151 iGPU. One stream, full
262,144 ctx (the model's native window, `PARALLEL_SLOTS=1`), DFlash2 spec
decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, depth 4,
`SPEC_DRAFT_N_MAX=4`), reasoning extracted to `reasoning_content`, and
**no vision projector**. It replaced the Kairic Edge ROCmFPX build (jcbtc
Qwen3.8-27B-IU4-Kairic-Edge), which had replaced the unsloth
Qwen3.8-Flash-Next UD-IQ4_XS.
`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 29.3 GiB,
sha256-verified) on port 7770 from the gfx1151 iGPU. One session, full
262,144 ctx (the model's native window, `SESSIONS=1`), DFlash2 spec decode
(the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's **adaptive** block
policy, `--draft-tokens 7`), greedy sampling, reasoning extracted to
`reasoning_content`, and **no vision projector**. It replaced the unsloth
llama.cpp b11030 prebuilt, which had replaced the Kairic Edge ROCmFPX
build. Cut over 2026-09-28.
The spec method is DFlash2 (block-diffusion drafter) at depth 4. The
drafter is loaded via `--model-draft`; the chain is `--spec-type
draft-dflash` with `--spec-draft-device ROCm0`. KV is **q8_0** (the
unsloth b11030 build does not support fp8 KV; q8_0 is the
highest-precision 8-bit option, ~8 GiB at 262k). The serve script
(`scripts/serve-frank-large.sh`) validates binary → main GGUF → drafter
and exits with a pointer message if any is missing, then `exec`s
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
falls back to the built-in `peg-native` template, which produces a
~73,600-token prompt for a ~250-token request).
**The runtime is built from source on the host** —
`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with
`scripts/build-gufo.sh` (`--verify` re-checks an installed tree). gufo ships
no prebuilt Linux binaries and the v0.1.0/v0.1.1 release assets are empty.
Unlike the retired llama.cpp prebuilt, which bundled its own ROCm, **gufo
links the host ROCm 7.2.4 through its RPATH**, so the HIP dev packages must
stay installed. Two host workarounds the build script encodes:
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
is not the budget). The 27B dense model (31.5 GiB) is mmap'd in with
`--n-gpu-layers 999`; budget at 262k ≈ 41-43 GiB (weights + KV + buffers),
leaving ~71 GiB of aperture headroom. The 8 GiB prompt-cache budget
(`CACHE_RAM=8192`) is the qualified config.
- **Ubuntu's `librocwmma-dev` must be absent.** It installs a truncated
rocWMMA header set into `/usr/include` that shadows AMD's complete one
and breaks the build with `'internal/accessors.hpp' file not found`. A
`-I` cannot fix it — in HIP mode the host pass searches the system
directories first. The package has no reverse dependencies.
- **`--gcc-install-dir` must name GCC 15.** Ubuntu 26.04 ships
`/usr/include/c++/16`, which the ROCm clang prefers; its `<atomic_wait.h>`
needs a `std::__detail::__notify_impl` symbol version the installed
`libstdc++.so.6` (GLIBCXX_3.4.35) does not export, and the link fails.
`gufo diagnose` is the preflight and must report `[PASS]` with `gfx1151`.
It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for
`serve llm`) after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`, which
matters because ComfyUI holds queues on the same device; four HIP processes
is the device ceiling.
**The client contract changed in three ways worth knowing.** There is no
`--chat-template-file`: the official Qwen Jinja is compiled in, so the
Qwen-Sharp 73k-token-prompt failure mode cannot recur. `reasoning_effort`
now accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap`
fold for frank-large is no longer *required* — it is kept deliberately, so
that removing it does not silently move every `high` role from `xhigh` to
`high` effort inside a runtime swap. And there is no KV-quantization or
prompt-cache-RAM flag; prompt reuse is on by default and `--cache-disk`
is opt-in and off.
**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables
(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5×
faster, decode +7 % at zero depth and +40 % at 4 users — but **−14 % at
65,536 tokens of depth and −24 % at 131,072**. On the 8-bit target the
crossover is ~32k of cached context, and frank-large runs one 262,144-token
conversation, so long agent sessions sit on the losing side of that line.
The Q4_K_XL target shows no such regression; revisiting the quantization is
a separate decision from the runtime. Peak memory is 50.58 GiB at the full
262,144 context against the iGPU's 114 GiB GTT aperture.
The serve script (`scripts/serve-frank-large.sh`) validates binary → main
GGUF → drafter and exits with a pointer message if any is missing, then
`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model
…--served-model-name frank-large --context 262144 --speculative dflash2 …`.
The deployer's `deploy_llm_large` gates on the **runtime binary** as well
as the payload, and returns before it stops anything.
Fetch: `scripts/download-frank-large.sh` (per-file, each file verified
against its declared byte count + sha256; `--verify-only` re-checks).
against its declared byte count + sha256; `--verify-only` re-checks). The
payload is unchanged by every frank-large runtime swap since 2026-09-21.
Runbook: `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md`.
Rollback: [[frank-large-kairic-edge-qwen38-27b]] (payload and runtime
intact on disk).
Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`.
Rollback: [[frank-large-unsloth-qwen38-27b-q8xl]] (the b11030 prebuilt and
the payload are both intact on disk).
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
@@ -771,7 +803,7 @@ Active:
| Service | Runbook |
|---------|---------|
| frank-large (stock llama.cpp Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md` |
| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
@@ -781,6 +813,7 @@ Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |