chore(llm): add dankstatus.sh — one verified parse path for Dankserver state

The ad-hoc status commands kept producing wrong readings, most seriously
reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free.
Cause: /proc/meminfo separates key from value with a TAB, so

  grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2

returns an empty field and awk prints 0.0. A false "host is out of memory"
reading is worse than no reading, because it invites an unnecessary abort.

dankstatus.sh parses on a colon split (tab-agnostic), cross-checks
MemAvailable against MemTotal before printing either, and refuses to print
a zero value. It also pins the rocm-smi CSV column order (device, total,
used) which was previously read swapped, and takes only the first line of
systemctl is-active/is-enabled, which emit more than one line here.

Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU)
as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing
the GTT aperture. The device is smaller than documented, not larger.

Tests: 346/346 pass (no test surface touched).
This commit is contained in:
Frank committed 2026-09-28 23:27:50 -07:00
1 parent b4b6777e49
commit 32abc8d543
8 files changed
+525 -146

No files matched your search

+53 -26
View File
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port | | Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------| |--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 | | Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 29.3 GiB) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, compiled-in Qwen Jinja, text only | 7770 | | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in Retired endpoints (`frank-npu`, and every prior runtime) are in
@@ -277,20 +277,30 @@ FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
projector's audio encoder (frank-small transcribes WAV verbatim even though projector's audio encoder (frank-small transcribes WAV verbatim even though
`audio` is omitted from its `input` list — OMP's schema rejects it). `audio` is omitted from its `input` list — OMP's schema rejects it).
### frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE) ### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)
**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a **gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a
vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 / vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 /
Radeon 8060S, `gfx1151`) — as a **root systemd unit** Radeon 8060S, `gfx1151`) — as a **root systemd unit**
(`scripts/frank-large.service`, `User=dank`), serving (`scripts/frank-large.service`, `User=dank`), serving
`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 29.3 GiB, `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s `UD-Q8_K_XL`
sha256-verified) on port 7770 from the gfx1151 iGPU. One session, full (27B dense, 29.3 GiB, sha256-verified; **abliterated/uncensored**, and under
262,144 ctx (the model's native window, `SESSIONS=1`), DFlash2 spec decode the *non-OSI* SWIFT OPEN LICENSE 1.0) on port 7770 from the gfx1151 iGPU.
(the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's **adaptive** block One session, full 262,144 ctx (the model's native window, `SESSIONS=1`),
policy, `--draft-tokens 7`), greedy sampling, reasoning extracted to DFlash2 spec decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's
`reasoning_content`, and **no vision projector**. It replaced the unsloth **adaptive** block policy, `--draft-tokens 7`), greedy sampling, reasoning
llama.cpp b11030 prebuilt, which had replaced the Kairic Edge ROCmFPX extracted to `reasoning_content`, and **no vision projector**.
build. Cut over 2026-09-28.
**Two swaps landed on 2026-09-28**, both same day: the engine (from the
unsloth llama.cpp b11030 prebuilt, which had replaced Kairic Edge ROCmFPX)
and then the payload (from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on
disk and the payload-only rollback). The two targets are structurally
interchangeable for gufo — same `qwen35` architecture, the same 866 tensors
in the same order, the same per-tensor Q8_0/F32/BF16 layout, the same MTP
head at `blk.64` — verified by parsing and diffing both GGUF headers before
either was loaded. **The DFlash2 drafter did not move**, so it has its own
`DRAFTER_DIR` and the deployer's stray-run sweep keys on the drafter's
filename, not the target's.
**The runtime is built from source on the host** — **The runtime is built from source on the host** —
`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with `/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with
@@ -316,15 +326,27 @@ It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for
matters because ComfyUI holds queues on the same device; four HIP processes matters because ComfyUI holds queues on the same device; four HIP processes
is the device ceiling. is the device ceiling.
**The client contract changed in three ways worth knowing.** There is no **The client contract changed, and the chat template is the sharp edge.**
`--chat-template-file`: the official Qwen Jinja is compiled in, so the **gufo has no Jinja engine at all** — `src/models/qwen/chat_template.cpp` is
Qwen-Sharp 73k-token-prompt failure mode cannot recur. `reasoning_effort` 587 lines of hand-written C++ that appends strings, with no
now accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap` `--chat-template-file`, no env override, and no template body to point at.
fold for frank-large is no longer *required* — it is kept deliberately, so `scripts/qwen-sharp-v22.4.0.jinja` therefore **cannot** be supplied to
that removing it does not silently move every `high` role from `xhigh` to frank-large: that is a port, not a config line. Given the choice on
`high` effort inside a runtime swap. And there is no KV-quantization or 2026-09-28 the decision was to ship on gufo's stock renderer and accept the
prompt-cache-RAM flag; prompt reuse is on by default and `--cache-disk` loss — the `terse` system-prompt injection, the in-band `<|think_*|>`
is opt-in and off. control, the `json|xml` `tool_call_format` switch, the tool-argument
truncation and `add_vision_id` prefixes are all gone. gufo's own renderer
does keep the effort instruction strings and `preserve_thinking`, which are
what the harness depends on. **This is the thing to watch**: the endpoint
now runs an *abliterated* model without the prompt that tells it to stop
waffling. The cheap fix, if verbosity shows up, is to send the `terse` text
as an ordinary system message from `models.yml` — not a template change.
`reasoning_effort` accepts the full OpenAI set (all six levels verified
200), so `models.yml`'s `reasoningEffortMap` fold is no longer *required* —
it is kept deliberately, so that removing it does not silently move every
`high` role from `xhigh` to `high` effort inside a swap. And there is no
KV-quantization or prompt-cache-RAM flag; prompt reuse is on by default and
`--cache-disk` is opt-in and off.
**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables **Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables
(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5× (vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5×
@@ -336,20 +358,25 @@ The Q4_K_XL target shows no such regression; revisiting the quantization is
a separate decision from the runtime. Peak memory is 50.58 GiB at the full a separate decision from the runtime. Peak memory is 50.58 GiB at the full
262,144 context against the iGPU's 114 GiB GTT aperture. 262,144 context against the iGPU's 114 GiB GTT aperture.
The serve script (`scripts/serve-frank-large.sh`) validates binary → main The serve script (`scripts/serve-frank-large.sh`) validates runtime → target
GGUF → drafter and exits with a pointer message if any is missing, then GGUF → drafter and exits with a pointer message if any is missing, then
`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model `exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model
…--served-model-name frank-large --context 262144 --speculative dflash2 …`. …--served-model-name frank-large --context 262144 --speculative dflash2 …`.
The deployer's `deploy_llm_large` gates on the **runtime binary** as well The deployer's `deploy_llm_large` gates on the **runtime binary** as well
as the payload, and returns before it stops anything. as both payload paths, and returns before it stops anything.
Fetch: `scripts/download-frank-large.sh` (per-file, each file verified Fetch: `scripts/download-frank-large-swift.sh` (the target; per-file byte
against its declared byte count + sha256; `--verify-only` re-checks). The count + sha256 against the Hub LFS oid, no auth header — the repo is public
payload is unchanged by every frank-large runtime swap since 2026-09-21. and a token on the curl command line is world-readable) and
`scripts/download-frank-large.sh` (the DFlash2 drafter, which did not move).
Both take `--verify-only`.
Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`. Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`.
Rollback: [[frank-large-unsloth-qwen38-27b-q8xl]] (the b11030 prebuilt and Rollback, two ways: **payload only** — repoint `MODEL_FILE` at
the payload are both intact on disk). `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`,
intact on disk. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]], the
b11030 prebuilt at `/home/dank/GitPackages/unsloth-llama-b11030/`; the only
way to get Qwen-Sharp back.
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env bash
# Single source of truth for Dankserver GLM-5.3 state.
#
# This exists because the ad-hoc status commands kept producing WRONG readings.
# The specific trap: /proc/meminfo separates key from value with a TAB, so the
# common idiom
# grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2
# yields an empty field (tab is not a space) and awk then prints 0.0 -- which
# reads as "host out of memory" when the host actually has 240 GiB free.
# Everything below parses on a colon split, which is tab-agnostic, and the
# memory figure is cross-checked before it is shown.
set -uo pipefail
MB=1048576
GLM_CTR="${GLM_CTR:-glm-w4a16}"
mem_field() { awk -v k="$1:" '$1==k {printf "%.1f", $2/'"$MB"'; exit}' /proc/meminfo; }
section() { printf '\n== %s ==\n' "$1"; }
section "time / host"
printf ' %s %s\n' "$(date +%H:%M:%S)" \
"$(awk -F: '/model name/{print $2; exit}' /proc/cpuinfo | sed 's/^ *//')"
section "memory (GiB)"
TOT=$(mem_field MemTotal); AVAIL=$(mem_field MemAvailable)
printf ' MemTotal %8s\n' "$TOT"
printf ' MemAvailable %8s\n' "$AVAIL"
printf ' Cached %8s\n' "$(mem_field Cached)"
awk -v t="$TOT" -v a="$AVAIL" 'BEGIN{
if (t+0 <= 0 || a+0 <= 0) { print " *** PARSE BUG: zero value ***"; exit }
if (a+0 > t+0) { print " *** PARSE BUG: available > total ***"; exit }
printf " check ok (%.0f%% free)\n", 100*a/t
}'
section "swap"
swapon --show=NAME,SIZE,USED --noheadings 2>/dev/null \
| awk '{printf " %-12s %6s used of %s\n",$1,$3,$2}' || echo " none"
section "gpu"
# CSV layout: device,VRAM Total Memory (B),VRAM Total Used Memory (B)
rocm-smi --showmeminfo vram --csv 2>/dev/null | awk -F, '
NR>1 && $1 ~ /^card[0-9]+$/ {
printf " %-8s %7.2f GiB used of %.2f\n", $1, $3/1073741824, $2/1073741824
}'
printf ' compute util: '
rocm-smi --showuse --csv 2>/dev/null | awk -F, 'NR>1 && $1 ~ /^card[0-9]+$/ {printf "%s=%s%% ", $1, $2}'
echo
section "disk"
df -h /home/dank 2>/dev/null | awk 'NR==2{printf " %s %s used %s free (%s)\n",$1,$3,$4,$5}'
section "llm containers"
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qE 'glm|frank'; then
podman ps -a --format '{{.Names}}|{{.Status}}' 2>/dev/null \
| awk -F'|' '$1 ~ /glm|frank/ {printf " %-18s %s\n",$1,$2}'
else
echo " none"
fi
section "frank-primary unit"
printf ' active: %s\n' "$(systemctl is-active frank-primary 2>/dev/null | head -1)"
printf ' enabled: %s\n' "$(systemctl is-enabled frank-primary 2>/dev/null | head -1)"
section "glm serve tail"
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then
podman logs "$GLM_CTR" 2>&1 | grep -viE 'multiproc_executor\.py:943' | tail -5 | sed 's/^/ /'
else
echo " (container $GLM_CTR not present)"
fi
section "done"
+13 -9
View File
@@ -1096,7 +1096,7 @@ deploy_anubis() {
# same as every cutover since 2026-09-16: nothing starts until 7770 is # same as every cutover since 2026-09-16: nothing starts until 7770 is
# released. # released.
LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}" LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}"
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf}" LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}" LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
retire_halogen_large_quadlet() { retire_halogen_large_quadlet() {
@@ -1151,15 +1151,19 @@ deploy_llm_large() {
retire_halogen_large_quadlet retire_halogen_large_quadlet
# On a re-deploy the predecessor is THIS unit: stop it before waiting on # On a re-deploy the predecessor is THIS unit: stop it before waiting on
# the port, or the wait deadlocks on the server it just stopped serving. # the port, or the wait deadlocks on the server it just stopped serving.
# Then sweep a stray manual run of the SAME model -- that is how a bench # Then sweep a stray manual run -- that is how a bench harness leaves the
# harness leaves the port busy, and the pattern matches the gufo command # port busy. Match the DRAFTER path, which every frank-large command
# line because it carries the model path. Match the model path, not a # line carries via --dflash-model and which is UNCHANGED by the 2026-09-28
# binary name: frank-small on this host serves a different GGUF over # payload swap: the target moved to a different repository while the
# llama-server and must not be killed. The pattern never appears in the # drafter stayed put, so keying on the target filename would have quietly
# deploying shell's own command line, so -f is safe here (it is not in # stopped matching. It is also the right key in principle — it names this
# `bash deploy-remote.sh`). # endpoint's speculative drafter, not a binary. Match that, not a binary
# name: frank-small on this host serves a different GGUF over llama-server
# with no DFlash2 drafter and must not be killed. The pattern never
# appears in the deploying shell's own command line, so -f is safe here
# (it is not in `bash deploy-remote.sh`).
run_privileged systemctl stop frank-large.service 2>/dev/null || true run_privileged systemctl stop frank-large.service 2>/dev/null || true
pkill -f 'Qwen3.8-27B-UD-Q8_K_XL' 2>/dev/null || true pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M' 2>/dev/null || true
# Nothing may start until the predecessor has released the port. On # Nothing may start until the predecessor has released the port. On
# 2026-09-16 the cutover failed exactly here: the old server logged # 2026-09-16 the cutover failed exactly here: the old server logged
# "Deactivated successfully", the next engine tried to bind 7770 a second # "Deactivated successfully", the next engine tried to bind 7770 a second
+16 -7
View File
@@ -1,5 +1,5 @@
# frank-large — gufo (gufo-org/gufo) serving # frank-large — gufo (gufo-org/gufo) serving
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU (gfx1151). # Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) on the Strix Halo iGPU.
# #
# These are the tunables read by scripts/serve-frank-large.sh. The flag # These are the tunables read by scripts/serve-frank-large.sh. The flag
# list is owned by the serve script; this file only sets fleet values. # list is owned by the serve script; this file only sets fleet values.
@@ -8,16 +8,25 @@
# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one # runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
# exception because the serve script itself consumes it for --host. # exception because the serve script itself consumes it for --host.
# The payload (scripts/download-frank-large.sh): the serve script refuses to # The target payload (scripts/download-frank-large-swift.sh): the serve
# start without the main GGUF and the DFlash2 drafter. Both files are the # script refuses to start without the main GGUF and the DFlash2 drafter.
# same ones the llama.cpp deployment used — the cutover changed the RUNTIME, # Swapped in from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two
# not the weights. # are structurally interchangeable for gufo (same qwen35 architecture, the
MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL # same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16
# layout, the same MTP head), so only this path changed.
MODEL_DIR=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL
MODEL_FILE=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
# The DFlash2 drafter is UNCHANGED by the model swap and still lives beside
# the previous target, so it carries its own directory. It has a sliding
# window of 2048, which is what keeps its KV small (~40 MiB).
DRAFTER_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
DRAFTER_FILE=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
# The gufo runtime, built from source by scripts/build-gufo.sh. # The gufo runtime, built from source by scripts/build-gufo.sh.
RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1 RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1
# Fleet endpoint: the same port and alias every prior frank-large served, so # Fleet endpoint: the same port and alias every frank-large has served, so
# no client, probe or models.yml entry changes address. # no client, probe or models.yml entry changes address.
ALIAS=frank-large ALIAS=frank-large
HOST=0.0.0.0 HOST=0.0.0.0
+46 -23
View File
@@ -1,36 +1,52 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on Dankclaw's # Serve Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) as frank-large on
# Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the inference # Dankclaw's Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the
# engine built specifically for this hardware. # inference engine built specifically for this hardware.
# #
# Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source # Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source
# against the host ROCm by scripts/build-gufo.sh. It replaces the stock # against the host ROCm by scripts/build-gufo.sh. It replaced the stock
# llama.cpp prebuilt (unsloth b11030) that served this endpoint from # llama.cpp prebuilt (unsloth b11030) on 2026-09-28; port, alias and client
# 2026-09-21; payload, port and alias are unchanged, so no client moved. # contract did not move.
#
# Payload: ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's
# UD-Q8_K_XL, fetched by scripts/download-frank-large-swift.sh. It replaced
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two are structurally
# interchangeable for this engine — same `qwen35` architecture, the same 866
# tensors in the same order, the same per-tensor Q8_0/F32/BF16 layout, the
# same MTP head at blk.64 — so the swap needed no engine change. The fine-tune
# is abliterated (uncensored) and carries the SWIFT OPEN LICENSE 1.0, which
# is not an OSI licence.
# #
# Shape: one session, full 262144 context (the model's native window). The # Shape: one session, full 262144 context (the model's native window). The
# 27B dense weights (29.30 GiB) + the DFlash2 drafter (1.07 GiB) + KV + # 27B dense weights (29.3 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
# compute buffers fit the iGPU's GTT aperture (~114 GiB); gufo's own # compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this
# Q8/AR table measures 50.58 GiB peak at this context capacity. # context measured 66.5 GiB of the 116.7 GiB device.
# #
# Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block # Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block
# policy, capped at DRAFT_TOKENS proposals per step. The drafter is the # policy, capped at DRAFT_TOKENS proposals per step. The drafter is the
# z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048 # z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048
# (small draft KV). # (small draft KV). It is UNCHANGED by the model swap and still lives beside
# the previous target, which is why DRAFTER_DIR is its own tunable.
# #
# Sampling: greedy (--temperature 0). That is what this endpoint has always # Sampling: greedy (--temperature 0) server-side, as every frank-large has
# served, and it keeps the swap to a pure runtime change; gufo's per-model # served. The harness overrides it per request via extraBody (1.0 / 0.95 /
# recommendation for Qwen3.8 thinking is sampled (1.0 / 0.95 / top-k 20) and # top-k 20).
# is a separate decision.
# #
# Template: gufo compiles the official Qwen Jinja into the binary and takes # Template: gufo compiles the official Qwen Jinja into the binary and takes
# no --chat-template-file. Its Qwen27B default is thinking ON at xhigh effort # NO --chat-template-file — and it is not a Jinja engine at all
# with prior reasoning preserved, which is the contract the fleet already # (src/models/qwen/chat_template.cpp is a hand-written C++ renderer), so the
# depends on, so --think auto + --preserve-thinking on is spelled out here # fleet's scripts/qwen-sharp-v22.4.0.jinja cannot be supplied to it. That law
# rather than left implicit. # was given up when this endpoint moved to gufo on 2026-09-28, by decision:
# the terse system-prompt injection, the in-band <|think_*|> control, the
# json|xml tool_call_format switch and the tool-argument truncation are all
# absent. gufo's own renderer keeps the effort instruction strings and
# preserve_thinking, which are the parts the harness depends on. Accepted
# 2026-09-28; recovering the terse prompt from the client side is the
# cheapest partial restore.
# #
# Text-only: gufo auto-discovers a mmproj beside the model and this payload # Text-only: gufo auto-discovers a mmproj beside the model and this payload
# has none, so the endpoint stays text-only. # has none (the base repo ships one separately, unused), so the endpoint
# stays text-only.
# #
# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides # Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
# it; anything not set there falls through to the defaults below. # it; anything not set there falls through to the defaults below.
@@ -46,7 +62,7 @@ fi
# --- defaults (used when the env file does not set them) ------------------ # --- defaults (used when the env file does not set them) ------------------
: "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}" : "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}"
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}" : "${MODEL_DIR:=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL}"
: "${ALIAS:=frank-large}" : "${ALIAS:=frank-large}"
: "${HOST:=0.0.0.0}" : "${HOST:=0.0.0.0}"
: "${PORT:=7770}" : "${PORT:=7770}"
@@ -56,8 +72,15 @@ fi
: "${DRAFT_POLICY:=adaptive}" : "${DRAFT_POLICY:=adaptive}"
GUFO_BIN="$RUNTIME_DIR/bin/gufo" GUFO_BIN="$RUNTIME_DIR/bin/gufo"
MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" # The target GGUF and the drafter are named explicitly because they do not
# share a directory any more: the target is a different repository's artifact
# and keeps its own long filename, while the DFlash2 drafter is unchanged and
# still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to
# the deployed payload, so the env file only has to state them when it moves.
: "${MODEL_FILE:=$MODEL_DIR/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
: "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
: "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
if [[ ! -x "$GUFO_BIN" ]]; then if [[ ! -x "$GUFO_BIN" ]]; then
echo "runtime not found at $GUFO_BIN" >&2 echo "runtime not found at $GUFO_BIN" >&2
@@ -66,7 +89,7 @@ if [[ ! -x "$GUFO_BIN" ]]; then
fi fi
for f in "$MODEL_FILE" "$DRAFTER_FILE"; do for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
if [[ ! -f "$f" ]]; then if [[ ! -f "$f" ]]; then
echo "missing asset $f (run scripts/download-frank-large.sh)" >&2 echo "missing asset $f (run the matching scripts/download-frank-large*.sh)" >&2
exit 1 exit 1
fi fi
done done
@@ -1,11 +1,11 @@
--- ---
type: reference type: reference
title: "gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)" title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the one-session / full-window shape, what changed in the client contract versus llama.cpp, the measured prefill/decode trade, and the failure modes worth knowing." description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
tags: tags:
- llm - llm
- gufo - gufo
- unsloth - swift-1.5
- dankclaw - dankclaw
- frank-large - frank-large
- strix-halo - strix-halo
@@ -16,13 +16,20 @@ tags:
timestamp: 2026-09-28 timestamp: 2026-09-28
--- ---
# frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) # frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2)
> **ACTIVE since 2026-09-28.** Replaces > **ACTIVE since 2026-09-28.** Replaces
> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth > [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
> b11030 prebuilt), which had replaced > b11030 prebuilt), which had replaced
> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same > [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
> alias (`frank-large`), **same weights**. > alias (`frank-large`).
>
> **The payload also changed on 2026-09-28**, later the same day: the target
> went from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` to
> `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s
> `UD-Q8_K_XL` — an abliterated (uncensored) fine-tune of the same base. See
> [Payload](#payload) and, for what that costs, [the template
> law](#the-template-law-qwen-sharp-is-gone).
## Identity ## Identity
@@ -93,38 +100,123 @@ architecture, driver and queue budget.
## Shape ## Shape
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — 27B dense, ### Payload
29.30 GiB. **Unchanged by the cutover**, still fetched and sha256-verified
by `scripts/download-frank-large.sh`. - **Model:** `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`,
file `UD-Q8_K_XL` — 27B dense, 31,457,990,272 bytes, fetched and
sha256-verified (`9076f99a…`, the Hub LFS oid) by
`scripts/download-frank-large-swift.sh`. It is an **abliterated
(uncensored)** fine-tune of Qwen3.8-27B. The repository also publishes an
`mmproj-BF16.gguf`, which is **not** used: this endpoint is text-only.
- **Licence:** the repo declares `other` — the **SWIFT OPEN LICENSE 1.0**,
which is *not* an OSI licence. That is a real constraint on redistribution,
noted here so nobody discovers it later. It is fine for a private fleet.
- **Replaced:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on disk and
still the rollback target for the payload.
The two targets are **structurally interchangeable for this engine**, which
is why the swap needed no engine change. Header-diffed directly:
| | unsloth | Swift-1.5 |
|---|---|---|
| `general.architecture` | `qwen35` | `qwen35` |
| tensors (count, names, order) | 866 | 866, **identical** |
| per-tensor `ggml_type` | Q8_0×453, F32×360, BF16×53 | **identical** |
| per-tensor dims | — | **0 mismatches** |
| MTP head | 4 tensors at `blk.64.nextn.*` | identical |
| geometry | 65 blk, 5120 embd, 17408 FFN, 24/4 heads | identical |
The two metadata differences that look alarming are both no-ops, and both
were confirmed against gufo's source rather than assumed:
- **`general.file_type` is 15 vs 7.** gufo's `gguf_reader.cpp` states it is
*"deliberately not consulted… The Unsloth UD-\*_K_XL artifacts this repo
targets carry a stale one — UD-Q8_K_XL records ftype 15 (Q4_K_M) … The
tensor table always is."* The file gufo already served lies about itself
here, so gufo reads the tensor table.
- **Swift adds `qwen35.attention.recurrent_layers`.** gufo never reads that
key anywhere in `src/`; layer kinds come from `full_attention_interval`,
which both files carry as `4` and which agrees with it.
### Serving shape
- **Context:** one session, full 262,144 tokens (the model's native window), - **Context:** one session, full 262,144 tokens (the model's native window),
`SESSIONS=1`. The fleet's single stream is unchanged. `SESSIONS=1`. The fleet's single stream is unchanged.
- **Spec method:** DFlash2, gufo's **adaptive** block policy - **Spec method:** DFlash2, gufo's **adaptive** block policy
(`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the same (`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the
z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB). **unchanged** z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB) — it did not
- **Sampling:** greedy (`--temperature 0`), as every prior frank-large move with the target, which is why `DRAFTER_DIR` is a separate tunable
served. gufo's per-model recommendation for Qwen3.8 thinking is *sampled* from `MODEL_DIR` and why the deployer's stray-run sweep keys on the
(1.0 / 0.95 / top-k 20) — that is a separate decision, deliberately not drafter's filename rather than the target's.
bundled into a runtime swap. - **Sampling:** greedy (`--temperature 0`) server-side, as every prior
frank-large served. The harness overrides per request via `extraBody`
(1.0 / 0.95 / top-k 20).
- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak - **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture. with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
### Measured at the 2026-09-28 cutover ### Measured at the 2026-09-28 cutover
Live on Dankclaw after `deploy.sh dankclaw --service frank-large`: Live on Dankclaw. The **target** numbers are the Swift-1.5 load test
(a standalone run under the production flags, service stopped first); the
**engine** numbers are from the gufo cutover earlier the same day and are
unchanged by the payload swap.
- **13 s** from `load_started` to `listening` (target 8 s, drafter 2 s, - **9.2 s** `load_started` → `load_completed` for the Swift target, against a
session 3 s), against a 20-minute `LLM_WAIT` budget — a redeploy no longer 20-minute `LLM_WAIT` budget. The unsloth target loaded in 13 s.
sits in a load wait. - **66.5 GiB** of the 116.7 GiB iGPU device at context capacity 262,144;
- **RSS 30.6 GiB**, VSZ 87.3 GiB; host `used` 81 GiB of 121 GiB, i.e. the host RSS 30.6 GiB. No loader warnings, no skipped sections.
same order as the llama.cpp deployment it replaced. - Contract battery, all green on the Swift target: `/v1/models` →
- A short-prompt live request returned `context_length 262144`, a chat completion carrying both `content` and
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2, max_inter_token;dur=166.9` `reasoning_content`, a `tool_calls` response with
(≈41 tok/s). Short-prompt decode is gufo's *winning* regime; see the depth `finish_reason: tool_calls`, SSE streaming, and a **two-turn tool loop**
table above before quoting it as the endpoint's number. (tool call → tool result → follow-up) with prior reasoning preserved.
- Contract battery, all green: `/health` 200, `/ready` 200, Arithmetic spot-check: "17*23" → `391`.
`/v1/models` → `frank-large` with `context_length 262144`, a chat - Behavioural spot-check that this is the abliterated model: asked for a
completion carrying both `content` and `reasoning_content`, a `tool_calls` two-sentence lockpicking scene, it wrote one without hedging or refusing.
response with `finish_reason: tool_calls`, and SSE streaming. - From the gufo cutover, on the unsloth target: a short-prompt live request
returned
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (≈41 tok/s).
Short-prompt decode is gufo's *winning* regime; see the depth table above
before quoting it as the endpoint's number.
## The template law: Qwen-Sharp is gone
`scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is
wired into `frank-primary` as `R9V_CHAT_TEMPLATE_FILE` and it was
`frank-large`'s `--chat-template-file` under llama.cpp. **frank-large can no
longer use it**, and the reason is stronger than "gufo lacks the flag":
> gufo has **no Jinja engine at all.** `src/models/qwen/chat_template.cpp`
> is 587 lines of hand-written C++ that appends strings to build ChatML.
> There is no `--chat-template-file`, no environment override, and no
> template body anywhere in the binary to point at. Qwen-Sharp is 453 lines
> of Jinja2 using `namespace()`, macros, `raise_exception` and image/video
> branches — supplying it would mean **reimplementing it in C++**, not
> handing gufo a file.
Given the choice on 2026-09-28, the decision was to **ship on gufo's stock
Jinja and accept the loss**. What that gives up, concretely:
| Qwen-Sharp feature | On gufo |
|---|---|
| `terse` system-prompt injection (the headline behaviour) | **gone** |
| in-band `<\|think_off\|>` / `<\|think_xhigh\|>` control | **gone** |
| `tool_call_format` `json` \| `xml` switch | **gone** |
| `max_tool_arg_chars` / `max_tool_response_chars` truncation | **gone** |
| `add_vision_id` → `Picture N:` prefixes | **gone** |
| effort instruction strings in the system prompt | kept (gufo has its own, same text) |
| `preserve_thinking` | kept (`--preserve-thinking on`) |
The `terse` injection is the one that matters most here, and it is worth
being blunt about the combination: this endpoint now runs an **abliterated**
model *without* the prompt that tells it to stop waffling. If verbosity
shows up, the cheapest fix is to send the `terse` text as an ordinary
**system message from the client** (`models.yml` / the harness) rather than
through a template — that recovers the behaviour without forking the
renderer. The full-fidelity options are reverting to llama.cpp or porting
Qwen-Sharp into the C++ renderer; both are worse trades today, and the
second is the same class of change that produced the 2026-09-21
"73,600-token prompt" incident.
## Client contract — what actually changed ## Client contract — what actually changed
@@ -134,21 +226,22 @@ auto-discovers a `mmproj` beside the model; this payload has none).
The differences worth knowing: The differences worth knowing:
- **No custom chat template.** gufo compiles the official Qwen Jinja into the - **No custom chat template — see
binary and takes no `--chat-template-file`. The Qwen-Sharp v22.4.0 file [The template law](#the-template-law-qwen-sharp-is-gone) above**, which is
this endpoint used with llama.cpp is **no longer in the command line** — the most consequential change on this endpoint. Thinking defaults to on at
the 2026-09-21 "73,600-token prompt" incident cannot recur, because there `xhigh` with prior reasoning preserved (`--think auto
is no fallback template to land in. Thinking defaults to on at `xhigh` --preserve-thinking on`, spelled out in the serve script).
with prior reasoning preserved (`--think auto --preserve-thinking on`,
spelled out in the serve script).
- **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`, - **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`,
`low`, `medium`, `high`, `xhigh`, `max`). Qwen-Sharp rejected anything `low`, `medium`, `high`, `xhigh`, `max`), all verified 200 against the
outside `{xhigh, medium, low}` with an HTTP 500, which is why `models.yml` live endpoint. Qwen-Sharp rejected anything outside `{xhigh, medium, low}`
still carries a `reasoningEffortMap` fold for this model. **The fold is with an HTTP 500, which is why `models.yml` still carries a
kept on purpose**: removing it would silently move every `high` role from `reasoningEffortMap` fold for this model. **The fold is kept on
`xhigh` to `high` effort in the same commit as the runtime swap. Drop it as purpose**: removing it would silently move every `high` role from `xhigh`
its own change once the effort curve is a decision rather than a side to `high` effort. Drop it as its own change.
effect. - **Every field the harness sends is accepted.** `top_k`, `min_p`,
`presence_penalty`, `repetition_penalty` and `preserve_thinking` are all
outside gufo's documented request subset, and all returned 200. They are
applied from `models.yml`'s `extraBody` unchanged.
- **No KV-cache quantization flag.** gufo picks its own KV dtype; there is - **No KV-cache quantization flag.** gufo picks its own KV dtype; there is
no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it. no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it.
Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in
@@ -156,8 +249,8 @@ The differences worth knowing:
- **Health/readiness split:** `GET /health` is process liveness and is what - **Health/readiness split:** `GET /health` is process liveness and is what
the deployer gates on; `GET /ready` is model/backend readiness. the deployer gates on; `GET /ready` is model/backend readiness.
`GET /metrics` is Prometheus-format. `GET /metrics` is Prometheus-format.
- **Long bodies:** `--max-request-bytes` defaults to 8 MiB and unknown
request fields are rejected rather than ignored. - **Long bodies:** `--max-request-bytes` defaults to 8 MiB.
## Failure modes worth knowing ## Failure modes worth knowing
@@ -169,26 +262,44 @@ The differences worth knowing:
`/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself `/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself
(`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device (`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device
time-slices instead of failing; four HIP processes is the device ceiling. time-slices instead of failing; four HIP processes is the device ceiling.
- **Load is seconds, not minutes** (8 s measured cold), so a redeploy no - **Load is seconds, not minutes** (9.2 s measured cold on the Swift
longer spends 20 minutes in `LLM_WAIT`. target), so a redeploy no longer spends 20 minutes in `LLM_WAIT`.
- **A payload is two files in two directories.** The target and the drafter
no longer live together, so "the payload is present" is not one check; the
deployer gates on both paths and the serve script refuses to start if
either is absent. A symlinked "one payload directory" would hide that.
- **A v0.1.x runtime is a moving target.** Rebuilds are not automatic — - **A v0.1.x runtime is a moving target.** Rebuilds are not automatic —
`RUNTIME_DIR` names a version, and the deployer will not silently `RUNTIME_DIR` names a version, and the deployer will not silently
substitute another. substitute another.
## Rollback ## Rollback
[[frank-large-unsloth-qwen38-27b-q8xl]] — the unsloth b11030 prebuilt is Two independent rollbacks, because two things changed on 2026-09-28:
still at `/home/dank/GitPackages/unsloth-llama-b11030/` and the payload is
the same directory the current runtime reads. Roll back by reverting 1. **Payload only** — repoint `MODEL_FILE`/`MODEL_DIR` in
`RUNTIME_DIR` (and the gufo-only tunables) in `scripts/frank-large.env`, `scripts/frank-large.env` at
restoring the serve script's llama.cpp command, and redeploying. `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`
and redeploy. That directory is untouched on disk and the drafter already
lives in it. This is the fast way back to the stock Qwen weights on the
gufo runtime.
2. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]]: the unsloth b11030
prebuilt is still at `/home/dank/GitPackages/unsloth-llama-b11030/`.
Revert `RUNTIME_DIR` and the gufo-only tunables, restore the serve
script's llama.cpp command, and redeploy. This is also the only way to
get `qwen-sharp-v22.4.0.jinja` back.
# Citations # Citations
- Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT) - Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT)
- Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo - Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo
- API contract: `docs/SERVER.md`; flags: `docs/CLI.md` - API contract: `docs/SERVER.md`; flags: `docs/CLI.md`; the C++ renderer
that rules out a Jinja template:
`src/models/qwen/chat_template.cpp` (587 lines)
- GGUF header evidence for the payload swap: both files' headers were parsed
and diffed directly (architecture, all 866 tensor names in order, per-tensor
`ggml_type` and dims) before either was loaded
- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`; - Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`;
tunables: `scripts/frank-large.env` tunables: `scripts/frank-large.env`
- Payload: `scripts/download-frank-large.sh` - Payload: `scripts/download-frank-large-swift.sh` (target),
`scripts/download-frank-large.sh` (DFlash2 drafter)
- Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]] - Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]]
+71
View File
@@ -2,6 +2,77 @@
## 2026-09-28 ## 2026-09-28
- **frank-large payload swapped to Swift-1.5 (uncensored)** (production
change, Dankclaw). Frank asked whether the UD Q8 of
`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` would load on
gufo, then said to do it. Port 7770, alias `frank-large`, one 262,144
session, the DFlash2 drafter and the client contract are unchanged; the
27B dense weights are an **abliterated (uncensored)** fine-tune of the
same base, under the **non-OSI SWIFT OPEN LICENSE 1.0**.
- **Compatibility was established from the GGUF headers, not from the
filename.** Both files were range-fetched and parsed: same
`general.architecture` (`qwen35`), same 866 tensors with **identical
names in identical order**, **zero** per-tensor `ggml_type` or dim
mismatches (Q8_0×453, F32×360, BF16×53), same MTP head at `blk.64`, same
geometry. The two scary metadata differences are both no-ops, confirmed
against gufo's source: `general.file_type` (15 vs 7) is *"deliberately
not consulted"* by `gguf_reader.cpp` — the file gufo already served
records 15 (Q4_K_M) while containing no Q4_K tensor — and Swift's extra
`qwen35.attention.recurrent_layers` key is **never read** by gufo
anywhere in `src/`; layer kinds come from `full_attention_interval`,
which both files carry as `4` and which agrees with it.
- **Fetched and verified:** 31,457,990,272 bytes, sha256 `9076f99a…`
matching the Hub LFS oid, via the new
`/scripts/download-frank-large-swift.sh`. It sends **no auth header** —
the repo is public and ungated, and a token on the curl command line is
world-readable via `/proc/<pid>/cmdline`. The anonymous transfer ran at
~1.8 MiB/s; resuming with the host's token through a **mode-600
`curl -K` config** took it to ~7 MiB/s. The download survived an ssh
drop by running `setsid nohup`, not under the session.
- **Load test, service stopped first** (the GPU policy): **9.2 s** from
`load_started` to `load_completed`, 66.5 GiB of the 116.7 GiB iGPU at
context capacity 262,144, no loader warnings. Contract battery green:
`context_length 262144`, `content` + `reasoning_content`,
`finish_reason: tool_calls`, SSE, and a **two-turn tool loop** with prior
reasoning preserved. Spot-checks: "17*23" → `391`; asked for a
lockpicking scene it wrote one without hedging, confirming the
abliteration.
- **THE TEMPLATE LAW IS GONE, by decision.** Frank asked to "use our chat
template"; gufo **cannot** take it. This is not a missing flag —
`src/models/qwen/chat_template.cpp` is 587 lines of hand-written C++
that appends strings, with no Jinja engine, no `--chat-template-file`,
no env override and no template body in the binary. Qwen-Sharp
v22.4.0 is 453 lines of Jinja2 with `namespace()`, macros and
`raise_exception`, so supplying it is a **port, not a config line**.
Frank chose to ship on gufo's stock renderer and accept the loss: the
`terse` system-prompt injection, in-band `<|think_*|>` control, the
`json|xml` `tool_call_format` switch, tool-argument truncation and
`add_vision_id` prefixes are all gone. gufo's renderer does keep the
effort instruction strings and `preserve_thinking`. **Worth watching:
an abliterated model now runs without the prompt that tells it to stop
waffling** — the cheap fix is to send the `terse` text as an ordinary
system message from `models.yml`, not to touch a template.
- **The payload is now two files in two directories.** The target moved
repo; the DFlash2 drafter did not move, so `DRAFTER_DIR` is its own
tunable. That broke the deployer's stray-run sweep, which keyed on the
*target* filename and would have silently stopped matching — a stray
run would then have held 7770 past the port gate. It now keys on the
drafter, which every frank-large command line carries via
`--dflash-model` and which did not move.
- **`/tests/frank_large_unit.bats`** — 20 assertions. The mutation pass
found **four real gaps** on the first run, all now fixed: two fetch
assertions were satisfied by a **header comment** rather than the URL
assignment, so repointing a repo left them green; and nothing bound the
env's `MODEL_FILE`/`DRAFTER_FILE` to the filenames the fetch scripts
actually download, so a re-upload or an env edit would break the
endpoint with a "missing asset" no test predicted. There is now an
explicit cross-file invariant. A second harness bug worth recording:
a `bats --filter` that matches no test reports zero `not ok`, which
looks exactly like a mutation that survived — the harness now asserts
the `1..N` count before believing a result. 16/16 mutations caught.
- Docs: runbook gained a **Payload** section (the header evidence),
**The template law** section and a two-way **Rollback**; AGENTS.md
fleet table, frank-large section and rollback.
- **frank-large runtime swapped to gufo** (production change, Dankclaw). - **frank-large runtime swapped to gufo** (production change, Dankclaw).
Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine
built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S,
+91 -28
View File
@@ -2,10 +2,12 @@
# Seam: the frank-large root unit on dankclaw. # Seam: the frank-large root unit on dankclaw.
# #
# Since 2026-09-28 this endpoint is served by gufo (gufo-org/gufo), the # Since 2026-09-28 this endpoint is served by gufo (gufo-org/gufo), the
# Strix Halo inference engine, built from the pinned v0.1.1 tag and serving # Strix Halo inference engine built from the pinned v0.1.1 tag, serving
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter. It replaced # Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) UD-Q8_K_XL with the z-lab
# the stock llama.cpp prebuilt (unsloth b11030) on the same port, the same # DFlash2 Q4_K_M drafter. Two changes landed on this endpoint that day: the
# alias and the same weights. # engine (from the stock llama.cpp prebuilt) and the payload (from
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL). Port, alias and client contract did
# not move.
# #
# Most of this seam runs the serve script against a stub engine and asserts # Most of this seam runs the serve script against a stub engine and asserts
# on the argv it actually execs, so what is pinned is the command the # on the argv it actually execs, so what is pinned is the command the
@@ -17,15 +19,19 @@ load helpers
ENV_FILE="$REPO_ROOT/scripts/frank-large.env" ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
UNIT="$REPO_ROOT/scripts/frank-large.service" UNIT="$REPO_ROOT/scripts/frank-large.service"
SERVE="$REPO_ROOT/scripts/serve-frank-large.sh" SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
FETCH="$REPO_ROOT/scripts/download-frank-large.sh" FETCH="$REPO_ROOT/scripts/download-frank-large-swift.sh"
DRAFTER_FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
BUILD="$REPO_ROOT/scripts/build-gufo.sh" BUILD="$REPO_ROOT/scripts/build-gufo.sh"
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
# The deployed target's filename, kept here so a rename in the env file and a
# rename in this stub cannot drift apart unnoticed.
TARGET_STUB="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
setup() { setup() {
STAGE="$(mktemp -d)" STAGE="$(mktemp -d)"
RUNTIME_DIR="$STAGE/gufo" RUNTIME_DIR="$STAGE/gufo"
MODEL_DIR="$STAGE/models" MODEL_DIR="$STAGE/models"
mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR" DRAFTER_DIR="$STAGE/draft"
mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR" "$DRAFTER_DIR"
# A stub engine: the serve script execs it, so $@ is exactly what gufo # A stub engine: the serve script execs it, so $@ is exactly what gufo
# would be invoked with. # would be invoked with.
cat > "$RUNTIME_DIR/bin/gufo" <<'STUB' cat > "$RUNTIME_DIR/bin/gufo" <<'STUB'
@@ -33,12 +39,17 @@ setup() {
printf '%s\n' "$@" printf '%s\n' "$@"
STUB STUB
chmod +x "$RUNTIME_DIR/bin/gufo" chmod +x "$RUNTIME_DIR/bin/gufo"
: > "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf" # The target and the drafter live in SEPARATE directories since the
: > "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" # 2026-09-28 payload swap: the target came from a different repository,
# the DFlash2 drafter did not move. A stub that colocated them would
# never notice DRAFTER_DIR going stale.
: > "$MODEL_DIR/$TARGET_STUB"
: > "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
STUB_ENV="$STAGE/stub.env" STUB_ENV="$STAGE/stub.env"
cat > "$STUB_ENV" <<EOF cat > "$STUB_ENV" <<EOF
RUNTIME_DIR=$RUNTIME_DIR RUNTIME_DIR=$RUNTIME_DIR
MODEL_DIR=$MODEL_DIR MODEL_DIR=$MODEL_DIR
DRAFTER_DIR=$DRAFTER_DIR
EOF EOF
} }
@@ -116,11 +127,17 @@ argv_value() {
} }
@test "the shipped env file, not just the defaults, serves the deployed shape" { @test "the shipped env file, not just the defaults, serves the deployed shape" {
# The defaults are a floor; systemd sources the env file. Only the two # The defaults are a floor; systemd sources the env file. Only the
# on-host paths are redirected at the stub, so every fleet value in the # on-host paths are redirected at the stub, so every fleet value in the
# shipped file is the one under test. # shipped file is the one under test. MODEL_FILE and DRAFTER_FILE are
# absolute in the shipped file, so they need redirecting too — a sed that
# only rewrote MODEL_DIR would leave them pointing at /home/dank and the
# test would pass for the wrong reason (or fail on a box that has them).
sed -e "s#^RUNTIME_DIR=.*#RUNTIME_DIR=$RUNTIME_DIR#" \ sed -e "s#^RUNTIME_DIR=.*#RUNTIME_DIR=$RUNTIME_DIR#" \
-e "s#^MODEL_DIR=.*#MODEL_DIR=$MODEL_DIR#" \ -e "s#^MODEL_DIR=.*#MODEL_DIR=$MODEL_DIR#" \
-e "s#^MODEL_FILE=.*#MODEL_FILE=$MODEL_DIR/$TARGET_STUB#" \
-e "s#^DRAFTER_DIR=.*#DRAFTER_DIR=$DRAFTER_DIR#" \
-e "s#^DRAFTER_FILE=.*#DRAFTER_FILE=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf#" \
"$ENV_FILE" > "$STAGE/shipped.env" "$ENV_FILE" > "$STAGE/shipped.env"
serve_argv "$STAGE/shipped.env" serve_argv "$STAGE/shipped.env"
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
@@ -132,6 +149,10 @@ argv_value() {
[ "$(argv_value --speculative)" = "dflash2" ] [ "$(argv_value --speculative)" = "dflash2" ]
[ "$(argv_value --draft-tokens)" -gt 0 ] [ "$(argv_value --draft-tokens)" -gt 0 ]
[ "$(argv_value --draft-policy)" = "adaptive" ] [ "$(argv_value --draft-policy)" = "adaptive" ]
# The shipped file must name the target the fleet actually serves, not a
# path that merely happens to exist.
[ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ]
[ "$(argv_value --dflash-model)" = "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" ]
} }
@test "the engine is pointed at the payload that is actually on disk" { @test "the engine is pointed at the payload that is actually on disk" {
@@ -173,30 +194,68 @@ argv_value() {
! argv_has serve ! argv_has serve
} }
@test "the serve script refuses a missing drafter with the fetch pointer" { @test "the serve script refuses a missing drafter with a fetch pointer" {
rm -f "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" # The drafter lives in its own directory now, so the target can be present
# and this still has to be caught.
rm -f "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
serve_argv serve_argv
[ "$status" -ne 0 ] [ "$status" -ne 0 ]
[[ "$output" == *"missing asset"* ]] [[ "$output" == *"missing asset"* ]]
[[ "$output" == *"download-frank-large.sh"* ]] [[ "$output" == *"Qwen3.8-27B-DFlash2-Q4_K_M.gguf"* ]]
[[ "$output" == *"download-frank-large"* ]]
! argv_has serve ! argv_has serve
} }
@test "the serve script refuses a missing target and names the file" { @test "the serve script refuses a missing target and names the file" {
rm -f "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf" rm -f "$MODEL_DIR/$TARGET_STUB"
serve_argv serve_argv
[ "$status" -ne 0 ] [ "$status" -ne 0 ]
[[ "$output" == *"missing asset"* ]] [[ "$output" == *"missing asset"* ]]
[[ "$output" == *"Qwen3.8-27B-UD-Q8_K_XL.gguf"* ]] [[ "$output" == *"$TARGET_STUB"* ]]
} }
@test "the payload is fetched resumably and verified against declared sizes + sha256" { @test "both payload files are fetched resumably and verified against declared sizes + sha256" {
grep -q 'unsloth/Qwen3.8-27B-GGUF' "$FETCH" # The target and the drafter come from DIFFERENT repositories, fetched by
grep -q 'z-lab/Qwen3.8-27B-DFlash2-GGUF' "$FETCH" # two scripts. Anchor the repo checks to the URL ASSIGNMENT, not to the
grep -q -- '-fL -C -' "$FETCH" # file: the same repo string also appears in each script's header comment,
# Each file's local sha256 is compared to its declared oid and a # so a plain grep stayed green after the URL was repointed at another
# mismatch fails the run. # repository -- the script would then download something the serve script
grep -q 'SHA256 MISMATCH' "$FETCH" # never asked for.
grep -qE '^URL="https://huggingface\.co/ajgazin/Swift-1\.5-Qwen3\.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/' "$FETCH"
grep -qE 'https://huggingface\.co/z-lab/Qwen3\.8-27B-DFlash2-GGUF/resolve/main/' "$DRAFTER_FETCH"
for f in "$FETCH" "$DRAFTER_FETCH"; do
# -f so an HTTP error is a failure rather than a small "artifact";
# -C - so an interrupted 29 GiB run resumes.
grep -q -- '-fL -C -' "$f"
# The size and the sha256 are compared after the transfer, not trusted.
grep -q 'SHA256 MISMATCH' "$f"
grep -q 'SIZE MISMATCH' "$f"
done
}
@test "the serve script and the fetch scripts agree on the payload filenames" {
# The serve script's MODEL_FILE and the fetch script's REL are two
# independent spellings of one filename, and a repo is not required to
# keep them in step: a re-upload under a new name, or an env edited
# without the fetch, leaves the endpoint unable to start with a
# "missing asset" that no test would have predicted. Bind them here.
local served fetched drafter_served drafter_fetched
served="$(basename "$(grep '^MODEL_FILE=' "$ENV_FILE" | cut -d= -f2-)")"
fetched="$(sed -n 's/^REL="\(.*\)"$/\1/p' "$FETCH")"
[ -n "$served" ]
[ "$served" = "$fetched" ]
drafter_served="$(basename "$(grep '^DRAFTER_FILE=' "$ENV_FILE" | cut -d= -f2-)")"
drafter_fetched="$(sed -n 's/^ *"\(Qwen3\.8-27B-DFlash2[^ ]*\.gguf\) .*/\1/p' "$DRAFTER_FETCH")"
[ -n "$drafter_served" ]
[ "$drafter_served" = "$drafter_fetched" ]
}
@test "the drafter is still fetched by the script that carries its Hub repo" {
# Guards a future "simplification" that folds the drafter into the Swift
# script and silently orphans the artifact the serve script still points
# at via DRAFTER_DIR.
grep -q 'Qwen3.8-27B-UD-Q8_K_XL' "$DRAFTER_FETCH"
! grep -q 'DFlash2' "$FETCH"
} }
@test "the build script refuses a source tree that is not the pinned commit" { @test "the build script refuses a source tree that is not the pinned commit" {
@@ -247,11 +306,15 @@ argv_value() {
[ "$bail" -lt "$stop" ] [ "$bail" -lt "$stop" ]
} }
@test "the stray-run sweep cannot take frank-small down" { @test "the stray-run sweep keys on the drafter and cannot take frank-small down" {
# frank-small on this host serves a different GGUF. The sweep must match # The sweep used to key on the TARGET filename, which the 2026-09-28
# the model path, not a binary name, or a frank-large deploy restarts the # payload swap moved to a different repository. It would have stopped
# 3090 endpoint. # matching the live process, and a stray run would then hold 7770 past the
grep -q "pkill -f 'Qwen3.8-27B-UD-Q8_K_XL'" "$REMOTE_DEPLOY" # port gate and fail the deploy. The drafter is the part that did NOT
# move, and every frank-large command line carries it via --dflash-model.
grep -q "pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M'" "$REMOTE_DEPLOY"
# frank-small serves a different GGUF on the 3090 with no DFlash2 drafter,
# so a binary-name sweep is the only thing that would endanger it.
local large local large
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")" large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
! grep -q 'pkill -x llama-server' <<<"$large" ! grep -q 'pkill -x llama-server' <<<"$large"