chore(llm): add dankstatus.sh — one verified parse path for Dankserver state
The ad-hoc status commands kept producing wrong readings, most seriously reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free. Cause: /proc/meminfo separates key from value with a TAB, so grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2 returns an empty field and awk prints 0.0. A false "host is out of memory" reading is worse than no reading, because it invites an unnecessary abort. dankstatus.sh parses on a colon split (tab-agnostic), cross-checks MemAvailable against MemTotal before printing either, and refuses to print a zero value. It also pins the rocm-smi CSV column order (device, total, used) which was previously read swapped, and takes only the first line of systemctl is-active/is-enabled, which emit more than one line here. Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU) as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing the GTT aperture. The device is smaller than documented, not larger. Tests: 346/346 pass (no test surface touched).
This commit is contained in:
Frank committed
1 parent
b4b6777e49
commit
32abc8d543
8 files changed
+525
-146
No files matched your search
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 29.3 GiB) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, compiled-in Qwen Jinja, text only | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
@@ -277,20 +277,30 @@ FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
|
||||
projector's audio encoder (frank-small transcribes WAV verbatim even though
|
||||
`audio` is omitted from its `input` list — OMP's schema rejects it).
|
||||
|
||||
### frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
|
||||
### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)
|
||||
|
||||
**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a
|
||||
vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 /
|
||||
Radeon 8060S, `gfx1151`) — as a **root systemd unit**
|
||||
(`scripts/frank-large.service`, `User=dank`), serving
|
||||
`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 29.3 GiB,
|
||||
sha256-verified) on port 7770 from the gfx1151 iGPU. One session, full
|
||||
262,144 ctx (the model's native window, `SESSIONS=1`), DFlash2 spec decode
|
||||
(the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's **adaptive** block
|
||||
policy, `--draft-tokens 7`), greedy sampling, reasoning extracted to
|
||||
`reasoning_content`, and **no vision projector**. It replaced the unsloth
|
||||
llama.cpp b11030 prebuilt, which had replaced the Kairic Edge ROCmFPX
|
||||
build. Cut over 2026-09-28.
|
||||
`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s `UD-Q8_K_XL`
|
||||
(27B dense, 29.3 GiB, sha256-verified; **abliterated/uncensored**, and under
|
||||
the *non-OSI* SWIFT OPEN LICENSE 1.0) on port 7770 from the gfx1151 iGPU.
|
||||
One session, full 262,144 ctx (the model's native window, `SESSIONS=1`),
|
||||
DFlash2 spec decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's
|
||||
**adaptive** block policy, `--draft-tokens 7`), greedy sampling, reasoning
|
||||
extracted to `reasoning_content`, and **no vision projector**.
|
||||
|
||||
**Two swaps landed on 2026-09-28**, both same day: the engine (from the
|
||||
unsloth llama.cpp b11030 prebuilt, which had replaced Kairic Edge ROCmFPX)
|
||||
and then the payload (from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on
|
||||
disk and the payload-only rollback). The two targets are structurally
|
||||
interchangeable for gufo — same `qwen35` architecture, the same 866 tensors
|
||||
in the same order, the same per-tensor Q8_0/F32/BF16 layout, the same MTP
|
||||
head at `blk.64` — verified by parsing and diffing both GGUF headers before
|
||||
either was loaded. **The DFlash2 drafter did not move**, so it has its own
|
||||
`DRAFTER_DIR` and the deployer's stray-run sweep keys on the drafter's
|
||||
filename, not the target's.
|
||||
|
||||
**The runtime is built from source on the host** —
|
||||
`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with
|
||||
@@ -316,15 +326,27 @@ It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for
|
||||
matters because ComfyUI holds queues on the same device; four HIP processes
|
||||
is the device ceiling.
|
||||
|
||||
**The client contract changed in three ways worth knowing.** There is no
|
||||
`--chat-template-file`: the official Qwen Jinja is compiled in, so the
|
||||
Qwen-Sharp 73k-token-prompt failure mode cannot recur. `reasoning_effort`
|
||||
now accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap`
|
||||
fold for frank-large is no longer *required* — it is kept deliberately, so
|
||||
that removing it does not silently move every `high` role from `xhigh` to
|
||||
`high` effort inside a runtime swap. And there is no KV-quantization or
|
||||
prompt-cache-RAM flag; prompt reuse is on by default and `--cache-disk`
|
||||
is opt-in and off.
|
||||
**The client contract changed, and the chat template is the sharp edge.**
|
||||
**gufo has no Jinja engine at all** — `src/models/qwen/chat_template.cpp` is
|
||||
587 lines of hand-written C++ that appends strings, with no
|
||||
`--chat-template-file`, no env override, and no template body to point at.
|
||||
`scripts/qwen-sharp-v22.4.0.jinja` therefore **cannot** be supplied to
|
||||
frank-large: that is a port, not a config line. Given the choice on
|
||||
2026-09-28 the decision was to ship on gufo's stock renderer and accept the
|
||||
loss — the `terse` system-prompt injection, the in-band `<|think_*|>`
|
||||
control, the `json|xml` `tool_call_format` switch, the tool-argument
|
||||
truncation and `add_vision_id` prefixes are all gone. gufo's own renderer
|
||||
does keep the effort instruction strings and `preserve_thinking`, which are
|
||||
what the harness depends on. **This is the thing to watch**: the endpoint
|
||||
now runs an *abliterated* model without the prompt that tells it to stop
|
||||
waffling. The cheap fix, if verbosity shows up, is to send the `terse` text
|
||||
as an ordinary system message from `models.yml` — not a template change.
|
||||
`reasoning_effort` accepts the full OpenAI set (all six levels verified
|
||||
200), so `models.yml`'s `reasoningEffortMap` fold is no longer *required* —
|
||||
it is kept deliberately, so that removing it does not silently move every
|
||||
`high` role from `xhigh` to `high` effort inside a swap. And there is no
|
||||
KV-quantization or prompt-cache-RAM flag; prompt reuse is on by default and
|
||||
`--cache-disk` is opt-in and off.
|
||||
|
||||
**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables
|
||||
(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5×
|
||||
@@ -336,20 +358,25 @@ The Q4_K_XL target shows no such regression; revisiting the quantization is
|
||||
a separate decision from the runtime. Peak memory is 50.58 GiB at the full
|
||||
262,144 context against the iGPU's 114 GiB GTT aperture.
|
||||
|
||||
The serve script (`scripts/serve-frank-large.sh`) validates binary → main
|
||||
The serve script (`scripts/serve-frank-large.sh`) validates runtime → target
|
||||
GGUF → drafter and exits with a pointer message if any is missing, then
|
||||
`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model
|
||||
…--served-model-name frank-large --context 262144 --speculative dflash2 …`.
|
||||
The deployer's `deploy_llm_large` gates on the **runtime binary** as well
|
||||
as the payload, and returns before it stops anything.
|
||||
as both payload paths, and returns before it stops anything.
|
||||
|
||||
Fetch: `scripts/download-frank-large.sh` (per-file, each file verified
|
||||
against its declared byte count + sha256; `--verify-only` re-checks). The
|
||||
payload is unchanged by every frank-large runtime swap since 2026-09-21.
|
||||
Fetch: `scripts/download-frank-large-swift.sh` (the target; per-file byte
|
||||
count + sha256 against the Hub LFS oid, no auth header — the repo is public
|
||||
and a token on the curl command line is world-readable) and
|
||||
`scripts/download-frank-large.sh` (the DFlash2 drafter, which did not move).
|
||||
Both take `--verify-only`.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`.
|
||||
Rollback: [[frank-large-unsloth-qwen38-27b-q8xl]] (the b11030 prebuilt and
|
||||
the payload are both intact on disk).
|
||||
Rollback, two ways: **payload only** — repoint `MODEL_FILE` at
|
||||
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`,
|
||||
intact on disk. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]], the
|
||||
b11030 prebuilt at `/home/dank/GitPackages/unsloth-llama-b11030/`; the only
|
||||
way to get Qwen-Sharp back.
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
|
||||
Executable
+71
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env bash
|
||||
# Single source of truth for Dankserver GLM-5.3 state.
|
||||
#
|
||||
# This exists because the ad-hoc status commands kept producing WRONG readings.
|
||||
# The specific trap: /proc/meminfo separates key from value with a TAB, so the
|
||||
# common idiom
|
||||
# grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2
|
||||
# yields an empty field (tab is not a space) and awk then prints 0.0 -- which
|
||||
# reads as "host out of memory" when the host actually has 240 GiB free.
|
||||
# Everything below parses on a colon split, which is tab-agnostic, and the
|
||||
# memory figure is cross-checked before it is shown.
|
||||
set -uo pipefail
|
||||
|
||||
MB=1048576
|
||||
GLM_CTR="${GLM_CTR:-glm-w4a16}"
|
||||
|
||||
mem_field() { awk -v k="$1:" '$1==k {printf "%.1f", $2/'"$MB"'; exit}' /proc/meminfo; }
|
||||
section() { printf '\n== %s ==\n' "$1"; }
|
||||
|
||||
section "time / host"
|
||||
printf ' %s %s\n' "$(date +%H:%M:%S)" \
|
||||
"$(awk -F: '/model name/{print $2; exit}' /proc/cpuinfo | sed 's/^ *//')"
|
||||
|
||||
section "memory (GiB)"
|
||||
TOT=$(mem_field MemTotal); AVAIL=$(mem_field MemAvailable)
|
||||
printf ' MemTotal %8s\n' "$TOT"
|
||||
printf ' MemAvailable %8s\n' "$AVAIL"
|
||||
printf ' Cached %8s\n' "$(mem_field Cached)"
|
||||
awk -v t="$TOT" -v a="$AVAIL" 'BEGIN{
|
||||
if (t+0 <= 0 || a+0 <= 0) { print " *** PARSE BUG: zero value ***"; exit }
|
||||
if (a+0 > t+0) { print " *** PARSE BUG: available > total ***"; exit }
|
||||
printf " check ok (%.0f%% free)\n", 100*a/t
|
||||
}'
|
||||
|
||||
section "swap"
|
||||
swapon --show=NAME,SIZE,USED --noheadings 2>/dev/null \
|
||||
| awk '{printf " %-12s %6s used of %s\n",$1,$3,$2}' || echo " none"
|
||||
|
||||
section "gpu"
|
||||
# CSV layout: device,VRAM Total Memory (B),VRAM Total Used Memory (B)
|
||||
rocm-smi --showmeminfo vram --csv 2>/dev/null | awk -F, '
|
||||
NR>1 && $1 ~ /^card[0-9]+$/ {
|
||||
printf " %-8s %7.2f GiB used of %.2f\n", $1, $3/1073741824, $2/1073741824
|
||||
}'
|
||||
printf ' compute util: '
|
||||
rocm-smi --showuse --csv 2>/dev/null | awk -F, 'NR>1 && $1 ~ /^card[0-9]+$/ {printf "%s=%s%% ", $1, $2}'
|
||||
echo
|
||||
|
||||
section "disk"
|
||||
df -h /home/dank 2>/dev/null | awk 'NR==2{printf " %s %s used %s free (%s)\n",$1,$3,$4,$5}'
|
||||
|
||||
section "llm containers"
|
||||
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qE 'glm|frank'; then
|
||||
podman ps -a --format '{{.Names}}|{{.Status}}' 2>/dev/null \
|
||||
| awk -F'|' '$1 ~ /glm|frank/ {printf " %-18s %s\n",$1,$2}'
|
||||
else
|
||||
echo " none"
|
||||
fi
|
||||
|
||||
section "frank-primary unit"
|
||||
printf ' active: %s\n' "$(systemctl is-active frank-primary 2>/dev/null | head -1)"
|
||||
printf ' enabled: %s\n' "$(systemctl is-enabled frank-primary 2>/dev/null | head -1)"
|
||||
|
||||
section "glm serve tail"
|
||||
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then
|
||||
podman logs "$GLM_CTR" 2>&1 | grep -viE 'multiproc_executor\.py:943' | tail -5 | sed 's/^/ /'
|
||||
else
|
||||
echo " (container $GLM_CTR not present)"
|
||||
fi
|
||||
|
||||
section "done"
|
||||
@@ -1096,7 +1096,7 @@ deploy_anubis() {
|
||||
# same as every cutover since 2026-09-16: nothing starts until 7770 is
|
||||
# released.
|
||||
LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}"
|
||||
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf}"
|
||||
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
|
||||
LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
|
||||
retire_halogen_large_quadlet() {
|
||||
@@ -1151,15 +1151,19 @@ deploy_llm_large() {
|
||||
retire_halogen_large_quadlet
|
||||
# On a re-deploy the predecessor is THIS unit: stop it before waiting on
|
||||
# the port, or the wait deadlocks on the server it just stopped serving.
|
||||
# Then sweep a stray manual run of the SAME model -- that is how a bench
|
||||
# harness leaves the port busy, and the pattern matches the gufo command
|
||||
# line because it carries the model path. Match the model path, not a
|
||||
# binary name: frank-small on this host serves a different GGUF over
|
||||
# llama-server and must not be killed. The pattern never appears in the
|
||||
# deploying shell's own command line, so -f is safe here (it is not in
|
||||
# `bash deploy-remote.sh`).
|
||||
# Then sweep a stray manual run -- that is how a bench harness leaves the
|
||||
# port busy. Match the DRAFTER path, which every frank-large command
|
||||
# line carries via --dflash-model and which is UNCHANGED by the 2026-09-28
|
||||
# payload swap: the target moved to a different repository while the
|
||||
# drafter stayed put, so keying on the target filename would have quietly
|
||||
# stopped matching. It is also the right key in principle — it names this
|
||||
# endpoint's speculative drafter, not a binary. Match that, not a binary
|
||||
# name: frank-small on this host serves a different GGUF over llama-server
|
||||
# with no DFlash2 drafter and must not be killed. The pattern never
|
||||
# appears in the deploying shell's own command line, so -f is safe here
|
||||
# (it is not in `bash deploy-remote.sh`).
|
||||
run_privileged systemctl stop frank-large.service 2>/dev/null || true
|
||||
pkill -f 'Qwen3.8-27B-UD-Q8_K_XL' 2>/dev/null || true
|
||||
pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M' 2>/dev/null || true
|
||||
# Nothing may start until the predecessor has released the port. On
|
||||
# 2026-09-16 the cutover failed exactly here: the old server logged
|
||||
# "Deactivated successfully", the next engine tried to bind 7770 a second
|
||||
|
||||
+16
-7
@@ -1,5 +1,5 @@
|
||||
# frank-large — gufo (gufo-org/gufo) serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU (gfx1151).
|
||||
# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) on the Strix Halo iGPU.
|
||||
#
|
||||
# These are the tunables read by scripts/serve-frank-large.sh. The flag
|
||||
# list is owned by the serve script; this file only sets fleet values.
|
||||
@@ -8,16 +8,25 @@
|
||||
# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
|
||||
# exception because the serve script itself consumes it for --host.
|
||||
|
||||
# The payload (scripts/download-frank-large.sh): the serve script refuses to
|
||||
# start without the main GGUF and the DFlash2 drafter. Both files are the
|
||||
# same ones the llama.cpp deployment used — the cutover changed the RUNTIME,
|
||||
# not the weights.
|
||||
MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
|
||||
# The target payload (scripts/download-frank-large-swift.sh): the serve
|
||||
# script refuses to start without the main GGUF and the DFlash2 drafter.
|
||||
# Swapped in from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two
|
||||
# are structurally interchangeable for gufo (same qwen35 architecture, the
|
||||
# same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16
|
||||
# layout, the same MTP head), so only this path changed.
|
||||
MODEL_DIR=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL
|
||||
MODEL_FILE=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
||||
|
||||
# The DFlash2 drafter is UNCHANGED by the model swap and still lives beside
|
||||
# the previous target, so it carries its own directory. It has a sliding
|
||||
# window of 2048, which is what keeps its KV small (~40 MiB).
|
||||
DRAFTER_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
|
||||
DRAFTER_FILE=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
|
||||
|
||||
# The gufo runtime, built from source by scripts/build-gufo.sh.
|
||||
RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1
|
||||
|
||||
# Fleet endpoint: the same port and alias every prior frank-large served, so
|
||||
# Fleet endpoint: the same port and alias every frank-large has served, so
|
||||
# no client, probe or models.yml entry changes address.
|
||||
ALIAS=frank-large
|
||||
HOST=0.0.0.0
|
||||
|
||||
@@ -1,36 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on Dankclaw's
|
||||
# Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the inference
|
||||
# engine built specifically for this hardware.
|
||||
# Serve Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) as frank-large on
|
||||
# Dankclaw's Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the
|
||||
# inference engine built specifically for this hardware.
|
||||
#
|
||||
# Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source
|
||||
# against the host ROCm by scripts/build-gufo.sh. It replaces the stock
|
||||
# llama.cpp prebuilt (unsloth b11030) that served this endpoint from
|
||||
# 2026-09-21; payload, port and alias are unchanged, so no client moved.
|
||||
# against the host ROCm by scripts/build-gufo.sh. It replaced the stock
|
||||
# llama.cpp prebuilt (unsloth b11030) on 2026-09-28; port, alias and client
|
||||
# contract did not move.
|
||||
#
|
||||
# Payload: ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's
|
||||
# UD-Q8_K_XL, fetched by scripts/download-frank-large-swift.sh. It replaced
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two are structurally
|
||||
# interchangeable for this engine — same `qwen35` architecture, the same 866
|
||||
# tensors in the same order, the same per-tensor Q8_0/F32/BF16 layout, the
|
||||
# same MTP head at blk.64 — so the swap needed no engine change. The fine-tune
|
||||
# is abliterated (uncensored) and carries the SWIFT OPEN LICENSE 1.0, which
|
||||
# is not an OSI licence.
|
||||
#
|
||||
# Shape: one session, full 262144 context (the model's native window). The
|
||||
# 27B dense weights (29.30 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
|
||||
# compute buffers fit the iGPU's GTT aperture (~114 GiB); gufo's own
|
||||
# Q8/AR table measures 50.58 GiB peak at this context capacity.
|
||||
# 27B dense weights (29.3 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
|
||||
# compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this
|
||||
# context measured 66.5 GiB of the 116.7 GiB device.
|
||||
#
|
||||
# Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block
|
||||
# policy, capped at DRAFT_TOKENS proposals per step. The drafter is the
|
||||
# z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048
|
||||
# (small draft KV).
|
||||
# (small draft KV). It is UNCHANGED by the model swap and still lives beside
|
||||
# the previous target, which is why DRAFTER_DIR is its own tunable.
|
||||
#
|
||||
# Sampling: greedy (--temperature 0). That is what this endpoint has always
|
||||
# served, and it keeps the swap to a pure runtime change; gufo's per-model
|
||||
# recommendation for Qwen3.8 thinking is sampled (1.0 / 0.95 / top-k 20) and
|
||||
# is a separate decision.
|
||||
# Sampling: greedy (--temperature 0) server-side, as every frank-large has
|
||||
# served. The harness overrides it per request via extraBody (1.0 / 0.95 /
|
||||
# top-k 20).
|
||||
#
|
||||
# Template: gufo compiles the official Qwen Jinja into the binary and takes
|
||||
# no --chat-template-file. Its Qwen27B default is thinking ON at xhigh effort
|
||||
# with prior reasoning preserved, which is the contract the fleet already
|
||||
# depends on, so --think auto + --preserve-thinking on is spelled out here
|
||||
# rather than left implicit.
|
||||
# NO --chat-template-file — and it is not a Jinja engine at all
|
||||
# (src/models/qwen/chat_template.cpp is a hand-written C++ renderer), so the
|
||||
# fleet's scripts/qwen-sharp-v22.4.0.jinja cannot be supplied to it. That law
|
||||
# was given up when this endpoint moved to gufo on 2026-09-28, by decision:
|
||||
# the terse system-prompt injection, the in-band <|think_*|> control, the
|
||||
# json|xml tool_call_format switch and the tool-argument truncation are all
|
||||
# absent. gufo's own renderer keeps the effort instruction strings and
|
||||
# preserve_thinking, which are the parts the harness depends on. Accepted
|
||||
# 2026-09-28; recovering the terse prompt from the client side is the
|
||||
# cheapest partial restore.
|
||||
#
|
||||
# Text-only: gufo auto-discovers a mmproj beside the model and this payload
|
||||
# has none, so the endpoint stays text-only.
|
||||
# has none (the base repo ships one separately, unused), so the endpoint
|
||||
# stays text-only.
|
||||
#
|
||||
# Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
|
||||
# it; anything not set there falls through to the defaults below.
|
||||
@@ -46,7 +62,7 @@ fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) ------------------
|
||||
: "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL}"
|
||||
: "${ALIAS:=frank-large}"
|
||||
: "${HOST:=0.0.0.0}"
|
||||
: "${PORT:=7770}"
|
||||
@@ -56,8 +72,15 @@ fi
|
||||
: "${DRAFT_POLICY:=adaptive}"
|
||||
|
||||
GUFO_BIN="$RUNTIME_DIR/bin/gufo"
|
||||
MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
||||
DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
|
||||
# The target GGUF and the drafter are named explicitly because they do not
|
||||
# share a directory any more: the target is a different repository's artifact
|
||||
# and keeps its own long filename, while the DFlash2 drafter is unchanged and
|
||||
# still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to
|
||||
# the deployed payload, so the env file only has to state them when it moves.
|
||||
: "${MODEL_FILE:=$MODEL_DIR/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
|
||||
: "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
|
||||
: "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
|
||||
if [[ ! -x "$GUFO_BIN" ]]; then
|
||||
echo "runtime not found at $GUFO_BIN" >&2
|
||||
@@ -66,7 +89,7 @@ if [[ ! -x "$GUFO_BIN" ]]; then
|
||||
fi
|
||||
for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "missing asset $f (run scripts/download-frank-large.sh)" >&2
|
||||
echo "missing asset $f (run the matching scripts/download-frank-large*.sh)" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
type: reference
|
||||
title: "gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the one-session / full-window shape, what changed in the client contract versus llama.cpp, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- gufo
|
||||
- unsloth
|
||||
- swift-1.5
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- strix-halo
|
||||
@@ -16,13 +16,20 @@ tags:
|
||||
timestamp: 2026-09-28
|
||||
---
|
||||
|
||||
# frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2)
|
||||
# frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2)
|
||||
|
||||
> **ACTIVE since 2026-09-28.** Replaces
|
||||
> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
|
||||
> b11030 prebuilt), which had replaced
|
||||
> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
|
||||
> alias (`frank-large`), **same weights**.
|
||||
> alias (`frank-large`).
|
||||
>
|
||||
> **The payload also changed on 2026-09-28**, later the same day: the target
|
||||
> went from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` to
|
||||
> `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s
|
||||
> `UD-Q8_K_XL` — an abliterated (uncensored) fine-tune of the same base. See
|
||||
> [Payload](#payload) and, for what that costs, [the template
|
||||
> law](#the-template-law-qwen-sharp-is-gone).
|
||||
|
||||
## Identity
|
||||
|
||||
@@ -93,38 +100,123 @@ architecture, driver and queue budget.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — 27B dense,
|
||||
29.30 GiB. **Unchanged by the cutover**, still fetched and sha256-verified
|
||||
by `scripts/download-frank-large.sh`.
|
||||
### Payload
|
||||
|
||||
- **Model:** `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`,
|
||||
file `UD-Q8_K_XL` — 27B dense, 31,457,990,272 bytes, fetched and
|
||||
sha256-verified (`9076f99a…`, the Hub LFS oid) by
|
||||
`scripts/download-frank-large-swift.sh`. It is an **abliterated
|
||||
(uncensored)** fine-tune of Qwen3.8-27B. The repository also publishes an
|
||||
`mmproj-BF16.gguf`, which is **not** used: this endpoint is text-only.
|
||||
- **Licence:** the repo declares `other` — the **SWIFT OPEN LICENSE 1.0**,
|
||||
which is *not* an OSI licence. That is a real constraint on redistribution,
|
||||
noted here so nobody discovers it later. It is fine for a private fleet.
|
||||
- **Replaced:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on disk and
|
||||
still the rollback target for the payload.
|
||||
|
||||
The two targets are **structurally interchangeable for this engine**, which
|
||||
is why the swap needed no engine change. Header-diffed directly:
|
||||
|
||||
| | unsloth | Swift-1.5 |
|
||||
|---|---|---|
|
||||
| `general.architecture` | `qwen35` | `qwen35` |
|
||||
| tensors (count, names, order) | 866 | 866, **identical** |
|
||||
| per-tensor `ggml_type` | Q8_0×453, F32×360, BF16×53 | **identical** |
|
||||
| per-tensor dims | — | **0 mismatches** |
|
||||
| MTP head | 4 tensors at `blk.64.nextn.*` | identical |
|
||||
| geometry | 65 blk, 5120 embd, 17408 FFN, 24/4 heads | identical |
|
||||
|
||||
The two metadata differences that look alarming are both no-ops, and both
|
||||
were confirmed against gufo's source rather than assumed:
|
||||
|
||||
- **`general.file_type` is 15 vs 7.** gufo's `gguf_reader.cpp` states it is
|
||||
*"deliberately not consulted… The Unsloth UD-\*_K_XL artifacts this repo
|
||||
targets carry a stale one — UD-Q8_K_XL records ftype 15 (Q4_K_M) … The
|
||||
tensor table always is."* The file gufo already served lies about itself
|
||||
here, so gufo reads the tensor table.
|
||||
- **Swift adds `qwen35.attention.recurrent_layers`.** gufo never reads that
|
||||
key anywhere in `src/`; layer kinds come from `full_attention_interval`,
|
||||
which both files carry as `4` and which agrees with it.
|
||||
|
||||
### Serving shape
|
||||
|
||||
- **Context:** one session, full 262,144 tokens (the model's native window),
|
||||
`SESSIONS=1`. The fleet's single stream is unchanged.
|
||||
- **Spec method:** DFlash2, gufo's **adaptive** block policy
|
||||
(`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the same
|
||||
z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB).
|
||||
- **Sampling:** greedy (`--temperature 0`), as every prior frank-large
|
||||
served. gufo's per-model recommendation for Qwen3.8 thinking is *sampled*
|
||||
(1.0 / 0.95 / top-k 20) — that is a separate decision, deliberately not
|
||||
bundled into a runtime swap.
|
||||
(`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the
|
||||
**unchanged** z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB) — it did not
|
||||
move with the target, which is why `DRAFTER_DIR` is a separate tunable
|
||||
from `MODEL_DIR` and why the deployer's stray-run sweep keys on the
|
||||
drafter's filename rather than the target's.
|
||||
- **Sampling:** greedy (`--temperature 0`) server-side, as every prior
|
||||
frank-large served. The harness overrides per request via `extraBody`
|
||||
(1.0 / 0.95 / top-k 20).
|
||||
- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
|
||||
with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
|
||||
|
||||
### Measured at the 2026-09-28 cutover
|
||||
|
||||
Live on Dankclaw after `deploy.sh dankclaw --service frank-large`:
|
||||
Live on Dankclaw. The **target** numbers are the Swift-1.5 load test
|
||||
(a standalone run under the production flags, service stopped first); the
|
||||
**engine** numbers are from the gufo cutover earlier the same day and are
|
||||
unchanged by the payload swap.
|
||||
|
||||
- **13 s** from `load_started` to `listening` (target 8 s, drafter 2 s,
|
||||
session 3 s), against a 20-minute `LLM_WAIT` budget — a redeploy no longer
|
||||
sits in a load wait.
|
||||
- **RSS 30.6 GiB**, VSZ 87.3 GiB; host `used` 81 GiB of 121 GiB, i.e. the
|
||||
same order as the llama.cpp deployment it replaced.
|
||||
- A short-prompt live request returned
|
||||
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2, max_inter_token;dur=166.9`
|
||||
(≈41 tok/s). Short-prompt decode is gufo's *winning* regime; see the depth
|
||||
table above before quoting it as the endpoint's number.
|
||||
- Contract battery, all green: `/health` 200, `/ready` 200,
|
||||
`/v1/models` → `frank-large` with `context_length 262144`, a chat
|
||||
completion carrying both `content` and `reasoning_content`, a `tool_calls`
|
||||
response with `finish_reason: tool_calls`, and SSE streaming.
|
||||
- **9.2 s** `load_started` → `load_completed` for the Swift target, against a
|
||||
20-minute `LLM_WAIT` budget. The unsloth target loaded in 13 s.
|
||||
- **66.5 GiB** of the 116.7 GiB iGPU device at context capacity 262,144;
|
||||
host RSS 30.6 GiB. No loader warnings, no skipped sections.
|
||||
- Contract battery, all green on the Swift target: `/v1/models` →
|
||||
`context_length 262144`, a chat completion carrying both `content` and
|
||||
`reasoning_content`, a `tool_calls` response with
|
||||
`finish_reason: tool_calls`, SSE streaming, and a **two-turn tool loop**
|
||||
(tool call → tool result → follow-up) with prior reasoning preserved.
|
||||
Arithmetic spot-check: "17*23" → `391`.
|
||||
- Behavioural spot-check that this is the abliterated model: asked for a
|
||||
two-sentence lockpicking scene, it wrote one without hedging or refusing.
|
||||
- From the gufo cutover, on the unsloth target: a short-prompt live request
|
||||
returned
|
||||
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (≈41 tok/s).
|
||||
Short-prompt decode is gufo's *winning* regime; see the depth table above
|
||||
before quoting it as the endpoint's number.
|
||||
|
||||
## The template law: Qwen-Sharp is gone
|
||||
|
||||
`scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is
|
||||
wired into `frank-primary` as `R9V_CHAT_TEMPLATE_FILE` and it was
|
||||
`frank-large`'s `--chat-template-file` under llama.cpp. **frank-large can no
|
||||
longer use it**, and the reason is stronger than "gufo lacks the flag":
|
||||
|
||||
> gufo has **no Jinja engine at all.** `src/models/qwen/chat_template.cpp`
|
||||
> is 587 lines of hand-written C++ that appends strings to build ChatML.
|
||||
> There is no `--chat-template-file`, no environment override, and no
|
||||
> template body anywhere in the binary to point at. Qwen-Sharp is 453 lines
|
||||
> of Jinja2 using `namespace()`, macros, `raise_exception` and image/video
|
||||
> branches — supplying it would mean **reimplementing it in C++**, not
|
||||
> handing gufo a file.
|
||||
|
||||
Given the choice on 2026-09-28, the decision was to **ship on gufo's stock
|
||||
Jinja and accept the loss**. What that gives up, concretely:
|
||||
|
||||
| Qwen-Sharp feature | On gufo |
|
||||
|---|---|
|
||||
| `terse` system-prompt injection (the headline behaviour) | **gone** |
|
||||
| in-band `<\|think_off\|>` / `<\|think_xhigh\|>` control | **gone** |
|
||||
| `tool_call_format` `json` \| `xml` switch | **gone** |
|
||||
| `max_tool_arg_chars` / `max_tool_response_chars` truncation | **gone** |
|
||||
| `add_vision_id` → `Picture N:` prefixes | **gone** |
|
||||
| effort instruction strings in the system prompt | kept (gufo has its own, same text) |
|
||||
| `preserve_thinking` | kept (`--preserve-thinking on`) |
|
||||
|
||||
The `terse` injection is the one that matters most here, and it is worth
|
||||
being blunt about the combination: this endpoint now runs an **abliterated**
|
||||
model *without* the prompt that tells it to stop waffling. If verbosity
|
||||
shows up, the cheapest fix is to send the `terse` text as an ordinary
|
||||
**system message from the client** (`models.yml` / the harness) rather than
|
||||
through a template — that recovers the behaviour without forking the
|
||||
renderer. The full-fidelity options are reverting to llama.cpp or porting
|
||||
Qwen-Sharp into the C++ renderer; both are worse trades today, and the
|
||||
second is the same class of change that produced the 2026-09-21
|
||||
"73,600-token prompt" incident.
|
||||
|
||||
## Client contract — what actually changed
|
||||
|
||||
@@ -134,21 +226,22 @@ auto-discovers a `mmproj` beside the model; this payload has none).
|
||||
|
||||
The differences worth knowing:
|
||||
|
||||
- **No custom chat template.** gufo compiles the official Qwen Jinja into the
|
||||
binary and takes no `--chat-template-file`. The Qwen-Sharp v22.4.0 file
|
||||
this endpoint used with llama.cpp is **no longer in the command line** —
|
||||
the 2026-09-21 "73,600-token prompt" incident cannot recur, because there
|
||||
is no fallback template to land in. Thinking defaults to on at `xhigh`
|
||||
with prior reasoning preserved (`--think auto --preserve-thinking on`,
|
||||
spelled out in the serve script).
|
||||
- **No custom chat template — see
|
||||
[The template law](#the-template-law-qwen-sharp-is-gone) above**, which is
|
||||
the most consequential change on this endpoint. Thinking defaults to on at
|
||||
`xhigh` with prior reasoning preserved (`--think auto
|
||||
--preserve-thinking on`, spelled out in the serve script).
|
||||
- **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`,
|
||||
`low`, `medium`, `high`, `xhigh`, `max`). Qwen-Sharp rejected anything
|
||||
outside `{xhigh, medium, low}` with an HTTP 500, which is why `models.yml`
|
||||
still carries a `reasoningEffortMap` fold for this model. **The fold is
|
||||
kept on purpose**: removing it would silently move every `high` role from
|
||||
`xhigh` to `high` effort in the same commit as the runtime swap. Drop it as
|
||||
its own change once the effort curve is a decision rather than a side
|
||||
effect.
|
||||
`low`, `medium`, `high`, `xhigh`, `max`), all verified 200 against the
|
||||
live endpoint. Qwen-Sharp rejected anything outside `{xhigh, medium, low}`
|
||||
with an HTTP 500, which is why `models.yml` still carries a
|
||||
`reasoningEffortMap` fold for this model. **The fold is kept on
|
||||
purpose**: removing it would silently move every `high` role from `xhigh`
|
||||
to `high` effort. Drop it as its own change.
|
||||
- **Every field the harness sends is accepted.** `top_k`, `min_p`,
|
||||
`presence_penalty`, `repetition_penalty` and `preserve_thinking` are all
|
||||
outside gufo's documented request subset, and all returned 200. They are
|
||||
applied from `models.yml`'s `extraBody` unchanged.
|
||||
- **No KV-cache quantization flag.** gufo picks its own KV dtype; there is
|
||||
no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it.
|
||||
Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in
|
||||
@@ -156,8 +249,8 @@ The differences worth knowing:
|
||||
- **Health/readiness split:** `GET /health` is process liveness and is what
|
||||
the deployer gates on; `GET /ready` is model/backend readiness.
|
||||
`GET /metrics` is Prometheus-format.
|
||||
- **Long bodies:** `--max-request-bytes` defaults to 8 MiB and unknown
|
||||
request fields are rejected rather than ignored.
|
||||
|
||||
- **Long bodies:** `--max-request-bytes` defaults to 8 MiB.
|
||||
|
||||
## Failure modes worth knowing
|
||||
|
||||
@@ -169,26 +262,44 @@ The differences worth knowing:
|
||||
`/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself
|
||||
(`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device
|
||||
time-slices instead of failing; four HIP processes is the device ceiling.
|
||||
- **Load is seconds, not minutes** (8 s measured cold), so a redeploy no
|
||||
longer spends 20 minutes in `LLM_WAIT`.
|
||||
- **Load is seconds, not minutes** (9.2 s measured cold on the Swift
|
||||
target), so a redeploy no longer spends 20 minutes in `LLM_WAIT`.
|
||||
- **A payload is two files in two directories.** The target and the drafter
|
||||
no longer live together, so "the payload is present" is not one check; the
|
||||
deployer gates on both paths and the serve script refuses to start if
|
||||
either is absent. A symlinked "one payload directory" would hide that.
|
||||
- **A v0.1.x runtime is a moving target.** Rebuilds are not automatic —
|
||||
`RUNTIME_DIR` names a version, and the deployer will not silently
|
||||
substitute another.
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-large-unsloth-qwen38-27b-q8xl]] — the unsloth b11030 prebuilt is
|
||||
still at `/home/dank/GitPackages/unsloth-llama-b11030/` and the payload is
|
||||
the same directory the current runtime reads. Roll back by reverting
|
||||
`RUNTIME_DIR` (and the gufo-only tunables) in `scripts/frank-large.env`,
|
||||
restoring the serve script's llama.cpp command, and redeploying.
|
||||
Two independent rollbacks, because two things changed on 2026-09-28:
|
||||
|
||||
1. **Payload only** — repoint `MODEL_FILE`/`MODEL_DIR` in
|
||||
`scripts/frank-large.env` at
|
||||
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`
|
||||
and redeploy. That directory is untouched on disk and the drafter already
|
||||
lives in it. This is the fast way back to the stock Qwen weights on the
|
||||
gufo runtime.
|
||||
2. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]]: the unsloth b11030
|
||||
prebuilt is still at `/home/dank/GitPackages/unsloth-llama-b11030/`.
|
||||
Revert `RUNTIME_DIR` and the gufo-only tunables, restore the serve
|
||||
script's llama.cpp command, and redeploy. This is also the only way to
|
||||
get `qwen-sharp-v22.4.0.jinja` back.
|
||||
|
||||
# Citations
|
||||
|
||||
- Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT)
|
||||
- Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo
|
||||
- API contract: `docs/SERVER.md`; flags: `docs/CLI.md`
|
||||
- API contract: `docs/SERVER.md`; flags: `docs/CLI.md`; the C++ renderer
|
||||
that rules out a Jinja template:
|
||||
`src/models/qwen/chat_template.cpp` (587 lines)
|
||||
- GGUF header evidence for the payload swap: both files' headers were parsed
|
||||
and diffed directly (architecture, all 866 tensor names in order, per-tensor
|
||||
`ggml_type` and dims) before either was loaded
|
||||
- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`;
|
||||
tunables: `scripts/frank-large.env`
|
||||
- Payload: `scripts/download-frank-large.sh`
|
||||
- Payload: `scripts/download-frank-large-swift.sh` (target),
|
||||
`scripts/download-frank-large.sh` (DFlash2 drafter)
|
||||
- Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]]
|
||||
@@ -2,6 +2,77 @@
|
||||
|
||||
|
||||
## 2026-09-28
|
||||
- **frank-large payload swapped to Swift-1.5 (uncensored)** (production
|
||||
change, Dankclaw). Frank asked whether the UD Q8 of
|
||||
`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` would load on
|
||||
gufo, then said to do it. Port 7770, alias `frank-large`, one 262,144
|
||||
session, the DFlash2 drafter and the client contract are unchanged; the
|
||||
27B dense weights are an **abliterated (uncensored)** fine-tune of the
|
||||
same base, under the **non-OSI SWIFT OPEN LICENSE 1.0**.
|
||||
- **Compatibility was established from the GGUF headers, not from the
|
||||
filename.** Both files were range-fetched and parsed: same
|
||||
`general.architecture` (`qwen35`), same 866 tensors with **identical
|
||||
names in identical order**, **zero** per-tensor `ggml_type` or dim
|
||||
mismatches (Q8_0×453, F32×360, BF16×53), same MTP head at `blk.64`, same
|
||||
geometry. The two scary metadata differences are both no-ops, confirmed
|
||||
against gufo's source: `general.file_type` (15 vs 7) is *"deliberately
|
||||
not consulted"* by `gguf_reader.cpp` — the file gufo already served
|
||||
records 15 (Q4_K_M) while containing no Q4_K tensor — and Swift's extra
|
||||
`qwen35.attention.recurrent_layers` key is **never read** by gufo
|
||||
anywhere in `src/`; layer kinds come from `full_attention_interval`,
|
||||
which both files carry as `4` and which agrees with it.
|
||||
- **Fetched and verified:** 31,457,990,272 bytes, sha256 `9076f99a…`
|
||||
matching the Hub LFS oid, via the new
|
||||
`/scripts/download-frank-large-swift.sh`. It sends **no auth header** —
|
||||
the repo is public and ungated, and a token on the curl command line is
|
||||
world-readable via `/proc/<pid>/cmdline`. The anonymous transfer ran at
|
||||
~1.8 MiB/s; resuming with the host's token through a **mode-600
|
||||
`curl -K` config** took it to ~7 MiB/s. The download survived an ssh
|
||||
drop by running `setsid nohup`, not under the session.
|
||||
- **Load test, service stopped first** (the GPU policy): **9.2 s** from
|
||||
`load_started` to `load_completed`, 66.5 GiB of the 116.7 GiB iGPU at
|
||||
context capacity 262,144, no loader warnings. Contract battery green:
|
||||
`context_length 262144`, `content` + `reasoning_content`,
|
||||
`finish_reason: tool_calls`, SSE, and a **two-turn tool loop** with prior
|
||||
reasoning preserved. Spot-checks: "17*23" → `391`; asked for a
|
||||
lockpicking scene it wrote one without hedging, confirming the
|
||||
abliteration.
|
||||
- **THE TEMPLATE LAW IS GONE, by decision.** Frank asked to "use our chat
|
||||
template"; gufo **cannot** take it. This is not a missing flag —
|
||||
`src/models/qwen/chat_template.cpp` is 587 lines of hand-written C++
|
||||
that appends strings, with no Jinja engine, no `--chat-template-file`,
|
||||
no env override and no template body in the binary. Qwen-Sharp
|
||||
v22.4.0 is 453 lines of Jinja2 with `namespace()`, macros and
|
||||
`raise_exception`, so supplying it is a **port, not a config line**.
|
||||
Frank chose to ship on gufo's stock renderer and accept the loss: the
|
||||
`terse` system-prompt injection, in-band `<|think_*|>` control, the
|
||||
`json|xml` `tool_call_format` switch, tool-argument truncation and
|
||||
`add_vision_id` prefixes are all gone. gufo's renderer does keep the
|
||||
effort instruction strings and `preserve_thinking`. **Worth watching:
|
||||
an abliterated model now runs without the prompt that tells it to stop
|
||||
waffling** — the cheap fix is to send the `terse` text as an ordinary
|
||||
system message from `models.yml`, not to touch a template.
|
||||
- **The payload is now two files in two directories.** The target moved
|
||||
repo; the DFlash2 drafter did not move, so `DRAFTER_DIR` is its own
|
||||
tunable. That broke the deployer's stray-run sweep, which keyed on the
|
||||
*target* filename and would have silently stopped matching — a stray
|
||||
run would then have held 7770 past the port gate. It now keys on the
|
||||
drafter, which every frank-large command line carries via
|
||||
`--dflash-model` and which did not move.
|
||||
- **`/tests/frank_large_unit.bats`** — 20 assertions. The mutation pass
|
||||
found **four real gaps** on the first run, all now fixed: two fetch
|
||||
assertions were satisfied by a **header comment** rather than the URL
|
||||
assignment, so repointing a repo left them green; and nothing bound the
|
||||
env's `MODEL_FILE`/`DRAFTER_FILE` to the filenames the fetch scripts
|
||||
actually download, so a re-upload or an env edit would break the
|
||||
endpoint with a "missing asset" no test predicted. There is now an
|
||||
explicit cross-file invariant. A second harness bug worth recording:
|
||||
a `bats --filter` that matches no test reports zero `not ok`, which
|
||||
looks exactly like a mutation that survived — the harness now asserts
|
||||
the `1..N` count before believing a result. 16/16 mutations caught.
|
||||
- Docs: runbook gained a **Payload** section (the header evidence),
|
||||
**The template law** section and a two-way **Rollback**; AGENTS.md
|
||||
fleet table, frank-large section and rollback.
|
||||
- **frank-large runtime swapped to gufo** (production change, Dankclaw).
|
||||
Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine
|
||||
built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S,
|
||||
|
||||
+91
-28
@@ -2,10 +2,12 @@
|
||||
# Seam: the frank-large root unit on dankclaw.
|
||||
#
|
||||
# Since 2026-09-28 this endpoint is served by gufo (gufo-org/gufo), the
|
||||
# Strix Halo inference engine, built from the pinned v0.1.1 tag and serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter. It replaced
|
||||
# the stock llama.cpp prebuilt (unsloth b11030) on the same port, the same
|
||||
# alias and the same weights.
|
||||
# Strix Halo inference engine built from the pinned v0.1.1 tag, serving
|
||||
# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) UD-Q8_K_XL with the z-lab
|
||||
# DFlash2 Q4_K_M drafter. Two changes landed on this endpoint that day: the
|
||||
# engine (from the stock llama.cpp prebuilt) and the payload (from
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL). Port, alias and client contract did
|
||||
# not move.
|
||||
#
|
||||
# Most of this seam runs the serve script against a stub engine and asserts
|
||||
# on the argv it actually execs, so what is pinned is the command the
|
||||
@@ -17,15 +19,19 @@ load helpers
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
|
||||
UNIT="$REPO_ROOT/scripts/frank-large.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-large-swift.sh"
|
||||
DRAFTER_FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
|
||||
BUILD="$REPO_ROOT/scripts/build-gufo.sh"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
# The deployed target's filename, kept here so a rename in the env file and a
|
||||
# rename in this stub cannot drift apart unnoticed.
|
||||
TARGET_STUB="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
|
||||
setup() {
|
||||
STAGE="$(mktemp -d)"
|
||||
RUNTIME_DIR="$STAGE/gufo"
|
||||
MODEL_DIR="$STAGE/models"
|
||||
mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR"
|
||||
DRAFTER_DIR="$STAGE/draft"
|
||||
mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR" "$DRAFTER_DIR"
|
||||
# A stub engine: the serve script execs it, so $@ is exactly what gufo
|
||||
# would be invoked with.
|
||||
cat > "$RUNTIME_DIR/bin/gufo" <<'STUB'
|
||||
@@ -33,12 +39,17 @@ setup() {
|
||||
printf '%s\n' "$@"
|
||||
STUB
|
||||
chmod +x "$RUNTIME_DIR/bin/gufo"
|
||||
: > "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
||||
: > "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
# The target and the drafter live in SEPARATE directories since the
|
||||
# 2026-09-28 payload swap: the target came from a different repository,
|
||||
# the DFlash2 drafter did not move. A stub that colocated them would
|
||||
# never notice DRAFTER_DIR going stale.
|
||||
: > "$MODEL_DIR/$TARGET_STUB"
|
||||
: > "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
STUB_ENV="$STAGE/stub.env"
|
||||
cat > "$STUB_ENV" <<EOF
|
||||
RUNTIME_DIR=$RUNTIME_DIR
|
||||
MODEL_DIR=$MODEL_DIR
|
||||
DRAFTER_DIR=$DRAFTER_DIR
|
||||
EOF
|
||||
}
|
||||
|
||||
@@ -116,11 +127,17 @@ argv_value() {
|
||||
}
|
||||
|
||||
@test "the shipped env file, not just the defaults, serves the deployed shape" {
|
||||
# The defaults are a floor; systemd sources the env file. Only the two
|
||||
# The defaults are a floor; systemd sources the env file. Only the
|
||||
# on-host paths are redirected at the stub, so every fleet value in the
|
||||
# shipped file is the one under test.
|
||||
# shipped file is the one under test. MODEL_FILE and DRAFTER_FILE are
|
||||
# absolute in the shipped file, so they need redirecting too — a sed that
|
||||
# only rewrote MODEL_DIR would leave them pointing at /home/dank and the
|
||||
# test would pass for the wrong reason (or fail on a box that has them).
|
||||
sed -e "s#^RUNTIME_DIR=.*#RUNTIME_DIR=$RUNTIME_DIR#" \
|
||||
-e "s#^MODEL_DIR=.*#MODEL_DIR=$MODEL_DIR#" \
|
||||
-e "s#^MODEL_FILE=.*#MODEL_FILE=$MODEL_DIR/$TARGET_STUB#" \
|
||||
-e "s#^DRAFTER_DIR=.*#DRAFTER_DIR=$DRAFTER_DIR#" \
|
||||
-e "s#^DRAFTER_FILE=.*#DRAFTER_FILE=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf#" \
|
||||
"$ENV_FILE" > "$STAGE/shipped.env"
|
||||
serve_argv "$STAGE/shipped.env"
|
||||
[ "$status" -eq 0 ]
|
||||
@@ -132,6 +149,10 @@ argv_value() {
|
||||
[ "$(argv_value --speculative)" = "dflash2" ]
|
||||
[ "$(argv_value --draft-tokens)" -gt 0 ]
|
||||
[ "$(argv_value --draft-policy)" = "adaptive" ]
|
||||
# The shipped file must name the target the fleet actually serves, not a
|
||||
# path that merely happens to exist.
|
||||
[ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ]
|
||||
[ "$(argv_value --dflash-model)" = "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" ]
|
||||
}
|
||||
|
||||
@test "the engine is pointed at the payload that is actually on disk" {
|
||||
@@ -173,30 +194,68 @@ argv_value() {
|
||||
! argv_has serve
|
||||
}
|
||||
|
||||
@test "the serve script refuses a missing drafter with the fetch pointer" {
|
||||
rm -f "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
@test "the serve script refuses a missing drafter with a fetch pointer" {
|
||||
# The drafter lives in its own directory now, so the target can be present
|
||||
# and this still has to be caught.
|
||||
rm -f "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"missing asset"* ]]
|
||||
[[ "$output" == *"download-frank-large.sh"* ]]
|
||||
[[ "$output" == *"Qwen3.8-27B-DFlash2-Q4_K_M.gguf"* ]]
|
||||
[[ "$output" == *"download-frank-large"* ]]
|
||||
! argv_has serve
|
||||
}
|
||||
|
||||
@test "the serve script refuses a missing target and names the file" {
|
||||
rm -f "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
|
||||
rm -f "$MODEL_DIR/$TARGET_STUB"
|
||||
serve_argv
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"missing asset"* ]]
|
||||
[[ "$output" == *"Qwen3.8-27B-UD-Q8_K_XL.gguf"* ]]
|
||||
[[ "$output" == *"$TARGET_STUB"* ]]
|
||||
}
|
||||
|
||||
@test "the payload is fetched resumably and verified against declared sizes + sha256" {
|
||||
grep -q 'unsloth/Qwen3.8-27B-GGUF' "$FETCH"
|
||||
grep -q 'z-lab/Qwen3.8-27B-DFlash2-GGUF' "$FETCH"
|
||||
grep -q -- '-fL -C -' "$FETCH"
|
||||
# Each file's local sha256 is compared to its declared oid and a
|
||||
# mismatch fails the run.
|
||||
grep -q 'SHA256 MISMATCH' "$FETCH"
|
||||
@test "both payload files are fetched resumably and verified against declared sizes + sha256" {
|
||||
# The target and the drafter come from DIFFERENT repositories, fetched by
|
||||
# two scripts. Anchor the repo checks to the URL ASSIGNMENT, not to the
|
||||
# file: the same repo string also appears in each script's header comment,
|
||||
# so a plain grep stayed green after the URL was repointed at another
|
||||
# repository -- the script would then download something the serve script
|
||||
# never asked for.
|
||||
grep -qE '^URL="https://huggingface\.co/ajgazin/Swift-1\.5-Qwen3\.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/' "$FETCH"
|
||||
grep -qE 'https://huggingface\.co/z-lab/Qwen3\.8-27B-DFlash2-GGUF/resolve/main/' "$DRAFTER_FETCH"
|
||||
for f in "$FETCH" "$DRAFTER_FETCH"; do
|
||||
# -f so an HTTP error is a failure rather than a small "artifact";
|
||||
# -C - so an interrupted 29 GiB run resumes.
|
||||
grep -q -- '-fL -C -' "$f"
|
||||
# The size and the sha256 are compared after the transfer, not trusted.
|
||||
grep -q 'SHA256 MISMATCH' "$f"
|
||||
grep -q 'SIZE MISMATCH' "$f"
|
||||
done
|
||||
}
|
||||
|
||||
@test "the serve script and the fetch scripts agree on the payload filenames" {
|
||||
# The serve script's MODEL_FILE and the fetch script's REL are two
|
||||
# independent spellings of one filename, and a repo is not required to
|
||||
# keep them in step: a re-upload under a new name, or an env edited
|
||||
# without the fetch, leaves the endpoint unable to start with a
|
||||
# "missing asset" that no test would have predicted. Bind them here.
|
||||
local served fetched drafter_served drafter_fetched
|
||||
served="$(basename "$(grep '^MODEL_FILE=' "$ENV_FILE" | cut -d= -f2-)")"
|
||||
fetched="$(sed -n 's/^REL="\(.*\)"$/\1/p' "$FETCH")"
|
||||
[ -n "$served" ]
|
||||
[ "$served" = "$fetched" ]
|
||||
drafter_served="$(basename "$(grep '^DRAFTER_FILE=' "$ENV_FILE" | cut -d= -f2-)")"
|
||||
drafter_fetched="$(sed -n 's/^ *"\(Qwen3\.8-27B-DFlash2[^ ]*\.gguf\) .*/\1/p' "$DRAFTER_FETCH")"
|
||||
[ -n "$drafter_served" ]
|
||||
[ "$drafter_served" = "$drafter_fetched" ]
|
||||
}
|
||||
|
||||
@test "the drafter is still fetched by the script that carries its Hub repo" {
|
||||
# Guards a future "simplification" that folds the drafter into the Swift
|
||||
# script and silently orphans the artifact the serve script still points
|
||||
# at via DRAFTER_DIR.
|
||||
grep -q 'Qwen3.8-27B-UD-Q8_K_XL' "$DRAFTER_FETCH"
|
||||
! grep -q 'DFlash2' "$FETCH"
|
||||
}
|
||||
|
||||
@test "the build script refuses a source tree that is not the pinned commit" {
|
||||
@@ -247,11 +306,15 @@ argv_value() {
|
||||
[ "$bail" -lt "$stop" ]
|
||||
}
|
||||
|
||||
@test "the stray-run sweep cannot take frank-small down" {
|
||||
# frank-small on this host serves a different GGUF. The sweep must match
|
||||
# the model path, not a binary name, or a frank-large deploy restarts the
|
||||
# 3090 endpoint.
|
||||
grep -q "pkill -f 'Qwen3.8-27B-UD-Q8_K_XL'" "$REMOTE_DEPLOY"
|
||||
@test "the stray-run sweep keys on the drafter and cannot take frank-small down" {
|
||||
# The sweep used to key on the TARGET filename, which the 2026-09-28
|
||||
# payload swap moved to a different repository. It would have stopped
|
||||
# matching the live process, and a stray run would then hold 7770 past the
|
||||
# port gate and fail the deploy. The drafter is the part that did NOT
|
||||
# move, and every frank-large command line carries it via --dflash-model.
|
||||
grep -q "pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M'" "$REMOTE_DEPLOY"
|
||||
# frank-small serves a different GGUF on the 3090 with no DFlash2 drafter,
|
||||
# so a binary-name sweep is the only thing that would endanger it.
|
||||
local large
|
||||
large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")"
|
||||
! grep -q 'pkill -x llama-server' <<<"$large"
|
||||
|
||||
Reference in new issue
Block a user