diff --git a/AGENTS.md b/AGENTS.md index 8cfdec5..70ad084 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 | -| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 29.3 GiB) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, compiled-in Qwen Jinja, text only | 7770 | +| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -277,20 +277,30 @@ FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4 projector's audio encoder (frank-small transcribes WAV verbatim even though `audio` is omitted from its `input` list — OMP's schema rejects it). -### frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE) +### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE) **gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, `gfx1151`) — as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`), serving -`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 29.3 GiB, -sha256-verified) on port 7770 from the gfx1151 iGPU. One session, full -262,144 ctx (the model's native window, `SESSIONS=1`), DFlash2 spec decode -(the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's **adaptive** block -policy, `--draft-tokens 7`), greedy sampling, reasoning extracted to -`reasoning_content`, and **no vision projector**. It replaced the unsloth -llama.cpp b11030 prebuilt, which had replaced the Kairic Edge ROCmFPX -build. Cut over 2026-09-28. +`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s `UD-Q8_K_XL` +(27B dense, 29.3 GiB, sha256-verified; **abliterated/uncensored**, and under +the *non-OSI* SWIFT OPEN LICENSE 1.0) on port 7770 from the gfx1151 iGPU. +One session, full 262,144 ctx (the model's native window, `SESSIONS=1`), +DFlash2 spec decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's +**adaptive** block policy, `--draft-tokens 7`), greedy sampling, reasoning +extracted to `reasoning_content`, and **no vision projector**. + +**Two swaps landed on 2026-09-28**, both same day: the engine (from the +unsloth llama.cpp b11030 prebuilt, which had replaced Kairic Edge ROCmFPX) +and then the payload (from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on +disk and the payload-only rollback). The two targets are structurally +interchangeable for gufo — same `qwen35` architecture, the same 866 tensors +in the same order, the same per-tensor Q8_0/F32/BF16 layout, the same MTP +head at `blk.64` — verified by parsing and diffing both GGUF headers before +either was loaded. **The DFlash2 drafter did not move**, so it has its own +`DRAFTER_DIR` and the deployer's stray-run sweep keys on the drafter's +filename, not the target's. **The runtime is built from source on the host** — `/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with @@ -316,15 +326,27 @@ It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for matters because ComfyUI holds queues on the same device; four HIP processes is the device ceiling. -**The client contract changed in three ways worth knowing.** There is no -`--chat-template-file`: the official Qwen Jinja is compiled in, so the -Qwen-Sharp 73k-token-prompt failure mode cannot recur. `reasoning_effort` -now accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap` -fold for frank-large is no longer *required* — it is kept deliberately, so -that removing it does not silently move every `high` role from `xhigh` to -`high` effort inside a runtime swap. And there is no KV-quantization or -prompt-cache-RAM flag; prompt reuse is on by default and `--cache-disk` -is opt-in and off. +**The client contract changed, and the chat template is the sharp edge.** +**gufo has no Jinja engine at all** — `src/models/qwen/chat_template.cpp` is +587 lines of hand-written C++ that appends strings, with no +`--chat-template-file`, no env override, and no template body to point at. +`scripts/qwen-sharp-v22.4.0.jinja` therefore **cannot** be supplied to +frank-large: that is a port, not a config line. Given the choice on +2026-09-28 the decision was to ship on gufo's stock renderer and accept the +loss — the `terse` system-prompt injection, the in-band `<|think_*|>` +control, the `json|xml` `tool_call_format` switch, the tool-argument +truncation and `add_vision_id` prefixes are all gone. gufo's own renderer +does keep the effort instruction strings and `preserve_thinking`, which are +what the harness depends on. **This is the thing to watch**: the endpoint +now runs an *abliterated* model without the prompt that tells it to stop +waffling. The cheap fix, if verbosity shows up, is to send the `terse` text +as an ordinary system message from `models.yml` — not a template change. +`reasoning_effort` accepts the full OpenAI set (all six levels verified +200), so `models.yml`'s `reasoningEffortMap` fold is no longer *required* — +it is kept deliberately, so that removing it does not silently move every +`high` role from `xhigh` to `high` effort inside a swap. And there is no +KV-quantization or prompt-cache-RAM flag; prompt reuse is on by default and +`--cache-disk` is opt-in and off. **Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables (vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5× @@ -336,20 +358,25 @@ The Q4_K_XL target shows no such regression; revisiting the quantization is a separate decision from the runtime. Peak memory is 50.58 GiB at the full 262,144 context against the iGPU's 114 GiB GTT aperture. -The serve script (`scripts/serve-frank-large.sh`) validates binary → main +The serve script (`scripts/serve-frank-large.sh`) validates runtime → target GGUF → drafter and exits with a pointer message if any is missing, then `exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model …--served-model-name frank-large --context 262144 --speculative dflash2 …`. The deployer's `deploy_llm_large` gates on the **runtime binary** as well -as the payload, and returns before it stops anything. +as both payload paths, and returns before it stops anything. -Fetch: `scripts/download-frank-large.sh` (per-file, each file verified -against its declared byte count + sha256; `--verify-only` re-checks). The -payload is unchanged by every frank-large runtime swap since 2026-09-21. +Fetch: `scripts/download-frank-large-swift.sh` (the target; per-file byte +count + sha256 against the Hub LFS oid, no auth header — the repo is public +and a token on the curl command line is world-readable) and +`scripts/download-frank-large.sh` (the DFlash2 drafter, which did not move). +Both take `--verify-only`. Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`. -Rollback: [[frank-large-unsloth-qwen38-27b-q8xl]] (the b11030 prebuilt and -the payload are both intact on disk). +Rollback, two ways: **payload only** — repoint `MODEL_FILE` at +`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`, +intact on disk. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]], the +b11030 prebuilt at `/home/dank/GitPackages/unsloth-llama-b11030/`; the only +way to get Qwen-Sharp back. ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) diff --git a/scripts/dankstatus.sh b/scripts/dankstatus.sh new file mode 100755 index 0000000..ca113b5 --- /dev/null +++ b/scripts/dankstatus.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# Single source of truth for Dankserver GLM-5.3 state. +# +# This exists because the ad-hoc status commands kept producing WRONG readings. +# The specific trap: /proc/meminfo separates key from value with a TAB, so the +# common idiom +# grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2 +# yields an empty field (tab is not a space) and awk then prints 0.0 -- which +# reads as "host out of memory" when the host actually has 240 GiB free. +# Everything below parses on a colon split, which is tab-agnostic, and the +# memory figure is cross-checked before it is shown. +set -uo pipefail + +MB=1048576 +GLM_CTR="${GLM_CTR:-glm-w4a16}" + +mem_field() { awk -v k="$1:" '$1==k {printf "%.1f", $2/'"$MB"'; exit}' /proc/meminfo; } +section() { printf '\n== %s ==\n' "$1"; } + +section "time / host" +printf ' %s %s\n' "$(date +%H:%M:%S)" \ + "$(awk -F: '/model name/{print $2; exit}' /proc/cpuinfo | sed 's/^ *//')" + +section "memory (GiB)" +TOT=$(mem_field MemTotal); AVAIL=$(mem_field MemAvailable) +printf ' MemTotal %8s\n' "$TOT" +printf ' MemAvailable %8s\n' "$AVAIL" +printf ' Cached %8s\n' "$(mem_field Cached)" +awk -v t="$TOT" -v a="$AVAIL" 'BEGIN{ + if (t+0 <= 0 || a+0 <= 0) { print " *** PARSE BUG: zero value ***"; exit } + if (a+0 > t+0) { print " *** PARSE BUG: available > total ***"; exit } + printf " check ok (%.0f%% free)\n", 100*a/t +}' + +section "swap" +swapon --show=NAME,SIZE,USED --noheadings 2>/dev/null \ + | awk '{printf " %-12s %6s used of %s\n",$1,$3,$2}' || echo " none" + +section "gpu" +# CSV layout: device,VRAM Total Memory (B),VRAM Total Used Memory (B) +rocm-smi --showmeminfo vram --csv 2>/dev/null | awk -F, ' + NR>1 && $1 ~ /^card[0-9]+$/ { + printf " %-8s %7.2f GiB used of %.2f\n", $1, $3/1073741824, $2/1073741824 + }' +printf ' compute util: ' +rocm-smi --showuse --csv 2>/dev/null | awk -F, 'NR>1 && $1 ~ /^card[0-9]+$/ {printf "%s=%s%% ", $1, $2}' +echo + +section "disk" +df -h /home/dank 2>/dev/null | awk 'NR==2{printf " %s %s used %s free (%s)\n",$1,$3,$4,$5}' + +section "llm containers" +if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qE 'glm|frank'; then + podman ps -a --format '{{.Names}}|{{.Status}}' 2>/dev/null \ + | awk -F'|' '$1 ~ /glm|frank/ {printf " %-18s %s\n",$1,$2}' +else + echo " none" +fi + +section "frank-primary unit" +printf ' active: %s\n' "$(systemctl is-active frank-primary 2>/dev/null | head -1)" +printf ' enabled: %s\n' "$(systemctl is-enabled frank-primary 2>/dev/null | head -1)" + +section "glm serve tail" +if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then + podman logs "$GLM_CTR" 2>&1 | grep -viE 'multiproc_executor\.py:943' | tail -5 | sed 's/^/ /' +else + echo " (container $GLM_CTR not present)" +fi + +section "done" diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 697b784..fa934c7 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1096,7 +1096,7 @@ deploy_anubis() { # same as every cutover since 2026-09-16: nothing starts until 7770 is # released. LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}" -LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf}" +LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}" LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}" retire_halogen_large_quadlet() { @@ -1151,15 +1151,19 @@ deploy_llm_large() { retire_halogen_large_quadlet # On a re-deploy the predecessor is THIS unit: stop it before waiting on # the port, or the wait deadlocks on the server it just stopped serving. - # Then sweep a stray manual run of the SAME model -- that is how a bench - # harness leaves the port busy, and the pattern matches the gufo command - # line because it carries the model path. Match the model path, not a - # binary name: frank-small on this host serves a different GGUF over - # llama-server and must not be killed. The pattern never appears in the - # deploying shell's own command line, so -f is safe here (it is not in - # `bash deploy-remote.sh`). + # Then sweep a stray manual run -- that is how a bench harness leaves the + # port busy. Match the DRAFTER path, which every frank-large command + # line carries via --dflash-model and which is UNCHANGED by the 2026-09-28 + # payload swap: the target moved to a different repository while the + # drafter stayed put, so keying on the target filename would have quietly + # stopped matching. It is also the right key in principle — it names this + # endpoint's speculative drafter, not a binary. Match that, not a binary + # name: frank-small on this host serves a different GGUF over llama-server + # with no DFlash2 drafter and must not be killed. The pattern never + # appears in the deploying shell's own command line, so -f is safe here + # (it is not in `bash deploy-remote.sh`). run_privileged systemctl stop frank-large.service 2>/dev/null || true - pkill -f 'Qwen3.8-27B-UD-Q8_K_XL' 2>/dev/null || true + pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M' 2>/dev/null || true # Nothing may start until the predecessor has released the port. On # 2026-09-16 the cutover failed exactly here: the old server logged # "Deactivated successfully", the next engine tried to bind 7770 a second diff --git a/scripts/frank-large.env b/scripts/frank-large.env index a1245f3..764b142 100644 --- a/scripts/frank-large.env +++ b/scripts/frank-large.env @@ -1,5 +1,5 @@ # frank-large — gufo (gufo-org/gufo) serving -# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU (gfx1151). +# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) on the Strix Halo iGPU. # # These are the tunables read by scripts/serve-frank-large.sh. The flag # list is owned by the serve script; this file only sets fleet values. @@ -8,16 +8,25 @@ # runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one # exception because the serve script itself consumes it for --host. -# The payload (scripts/download-frank-large.sh): the serve script refuses to -# start without the main GGUF and the DFlash2 drafter. Both files are the -# same ones the llama.cpp deployment used — the cutover changed the RUNTIME, -# not the weights. -MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL +# The target payload (scripts/download-frank-large-swift.sh): the serve +# script refuses to start without the main GGUF and the DFlash2 drafter. +# Swapped in from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two +# are structurally interchangeable for gufo (same qwen35 architecture, the +# same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16 +# layout, the same MTP head), so only this path changed. +MODEL_DIR=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL +MODEL_FILE=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf + +# The DFlash2 drafter is UNCHANGED by the model swap and still lives beside +# the previous target, so it carries its own directory. It has a sliding +# window of 2048, which is what keeps its KV small (~40 MiB). +DRAFTER_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL +DRAFTER_FILE=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf # The gufo runtime, built from source by scripts/build-gufo.sh. RUNTIME_DIR=/home/dank/GitPackages/gufo-v0.1.1 -# Fleet endpoint: the same port and alias every prior frank-large served, so +# Fleet endpoint: the same port and alias every frank-large has served, so # no client, probe or models.yml entry changes address. ALIAS=frank-large HOST=0.0.0.0 diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 2e4005d..578a20d 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -1,36 +1,52 @@ #!/usr/bin/env bash -# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on Dankclaw's -# Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the inference -# engine built specifically for this hardware. +# Serve Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) as frank-large on +# Dankclaw's Strix Halo iGPU (gfx1151), using gufo (gufo-org/gufo) — the +# inference engine built specifically for this hardware. # # Runtime: /home/dank/GitPackages/gufo-v0.1.1/bin/gufo, built from source -# against the host ROCm by scripts/build-gufo.sh. It replaces the stock -# llama.cpp prebuilt (unsloth b11030) that served this endpoint from -# 2026-09-21; payload, port and alias are unchanged, so no client moved. +# against the host ROCm by scripts/build-gufo.sh. It replaced the stock +# llama.cpp prebuilt (unsloth b11030) on 2026-09-28; port, alias and client +# contract did not move. +# +# Payload: ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's +# UD-Q8_K_XL, fetched by scripts/download-frank-large-swift.sh. It replaced +# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two are structurally +# interchangeable for this engine — same `qwen35` architecture, the same 866 +# tensors in the same order, the same per-tensor Q8_0/F32/BF16 layout, the +# same MTP head at blk.64 — so the swap needed no engine change. The fine-tune +# is abliterated (uncensored) and carries the SWIFT OPEN LICENSE 1.0, which +# is not an OSI licence. # # Shape: one session, full 262144 context (the model's native window). The -# 27B dense weights (29.30 GiB) + the DFlash2 drafter (1.07 GiB) + KV + -# compute buffers fit the iGPU's GTT aperture (~114 GiB); gufo's own -# Q8/AR table measures 50.58 GiB peak at this context capacity. +# 27B dense weights (29.3 GiB) + the DFlash2 drafter (1.07 GiB) + KV + +# compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this +# context measured 66.5 GiB of the 116.7 GiB device. # # Spec method: DFlash2 (a block-diffusion drafter), gufo's adaptive block # policy, capped at DRAFT_TOKENS proposals per step. The drafter is the # z-lab Qwen3.8-27B-DFlash2-Q4_K_M and it has a sliding window of 2048 -# (small draft KV). +# (small draft KV). It is UNCHANGED by the model swap and still lives beside +# the previous target, which is why DRAFTER_DIR is its own tunable. # -# Sampling: greedy (--temperature 0). That is what this endpoint has always -# served, and it keeps the swap to a pure runtime change; gufo's per-model -# recommendation for Qwen3.8 thinking is sampled (1.0 / 0.95 / top-k 20) and -# is a separate decision. +# Sampling: greedy (--temperature 0) server-side, as every frank-large has +# served. The harness overrides it per request via extraBody (1.0 / 0.95 / +# top-k 20). # # Template: gufo compiles the official Qwen Jinja into the binary and takes -# no --chat-template-file. Its Qwen27B default is thinking ON at xhigh effort -# with prior reasoning preserved, which is the contract the fleet already -# depends on, so --think auto + --preserve-thinking on is spelled out here -# rather than left implicit. +# NO --chat-template-file — and it is not a Jinja engine at all +# (src/models/qwen/chat_template.cpp is a hand-written C++ renderer), so the +# fleet's scripts/qwen-sharp-v22.4.0.jinja cannot be supplied to it. That law +# was given up when this endpoint moved to gufo on 2026-09-28, by decision: +# the terse system-prompt injection, the in-band <|think_*|> control, the +# json|xml tool_call_format switch and the tool-argument truncation are all +# absent. gufo's own renderer keeps the effort instruction strings and +# preserve_thinking, which are the parts the harness depends on. Accepted +# 2026-09-28; recovering the terse prompt from the client side is the +# cheapest partial restore. # # Text-only: gufo auto-discovers a mmproj beside the model and this payload -# has none, so the endpoint stays text-only. +# has none (the base repo ships one separately, unused), so the endpoint +# stays text-only. # # Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides # it; anything not set there falls through to the defaults below. @@ -46,7 +62,7 @@ fi # --- defaults (used when the env file does not set them) ------------------ : "${RUNTIME_DIR:=/home/dank/GitPackages/gufo-v0.1.1}" -: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}" +: "${MODEL_DIR:=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL}" : "${ALIAS:=frank-large}" : "${HOST:=0.0.0.0}" : "${PORT:=7770}" @@ -56,8 +72,15 @@ fi : "${DRAFT_POLICY:=adaptive}" GUFO_BIN="$RUNTIME_DIR/bin/gufo" -MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf" -DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" + +# The target GGUF and the drafter are named explicitly because they do not +# share a directory any more: the target is a different repository's artifact +# and keeps its own long filename, while the DFlash2 drafter is unchanged and +# still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to +# the deployed payload, so the env file only has to state them when it moves. +: "${MODEL_FILE:=$MODEL_DIR/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}" +: "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}" +: "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}" if [[ ! -x "$GUFO_BIN" ]]; then echo "runtime not found at $GUFO_BIN" >&2 @@ -66,7 +89,7 @@ if [[ ! -x "$GUFO_BIN" ]]; then fi for f in "$MODEL_FILE" "$DRAFTER_FILE"; do if [[ ! -f "$f" ]]; then - echo "missing asset $f (run scripts/download-frank-large.sh)" >&2 + echo "missing asset $f (run the matching scripts/download-frank-large*.sh)" >&2 exit 1 fi done diff --git a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md index 20864f9..5cac30e 100644 --- a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md +++ b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md @@ -1,11 +1,11 @@ --- type: reference -title: "gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)" -description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the one-session / full-window shape, what changed in the client contract versus llama.cpp, the measured prefill/decode trade, and the failure modes worth knowing." +title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)" +description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing." tags: - llm - gufo - - unsloth + - swift-1.5 - dankclaw - frank-large - strix-halo @@ -16,13 +16,20 @@ tags: timestamp: 2026-09-28 --- -# frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) +# frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) > **ACTIVE since 2026-09-28.** Replaces > [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth > b11030 prebuilt), which had replaced > [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same -> alias (`frank-large`), **same weights**. +> alias (`frank-large`). +> +> **The payload also changed on 2026-09-28**, later the same day: the target +> went from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` to +> `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s +> `UD-Q8_K_XL` — an abliterated (uncensored) fine-tune of the same base. See +> [Payload](#payload) and, for what that costs, [the template +> law](#the-template-law-qwen-sharp-is-gone). ## Identity @@ -93,38 +100,123 @@ architecture, driver and queue budget. ## Shape -- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — 27B dense, - 29.30 GiB. **Unchanged by the cutover**, still fetched and sha256-verified - by `scripts/download-frank-large.sh`. +### Payload + +- **Model:** `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`, + file `UD-Q8_K_XL` — 27B dense, 31,457,990,272 bytes, fetched and + sha256-verified (`9076f99a…`, the Hub LFS oid) by + `scripts/download-frank-large-swift.sh`. It is an **abliterated + (uncensored)** fine-tune of Qwen3.8-27B. The repository also publishes an + `mmproj-BF16.gguf`, which is **not** used: this endpoint is text-only. +- **Licence:** the repo declares `other` — the **SWIFT OPEN LICENSE 1.0**, + which is *not* an OSI licence. That is a real constraint on redistribution, + noted here so nobody discovers it later. It is fine for a private fleet. +- **Replaced:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on disk and + still the rollback target for the payload. + +The two targets are **structurally interchangeable for this engine**, which +is why the swap needed no engine change. Header-diffed directly: + +| | unsloth | Swift-1.5 | +|---|---|---| +| `general.architecture` | `qwen35` | `qwen35` | +| tensors (count, names, order) | 866 | 866, **identical** | +| per-tensor `ggml_type` | Q8_0×453, F32×360, BF16×53 | **identical** | +| per-tensor dims | — | **0 mismatches** | +| MTP head | 4 tensors at `blk.64.nextn.*` | identical | +| geometry | 65 blk, 5120 embd, 17408 FFN, 24/4 heads | identical | + +The two metadata differences that look alarming are both no-ops, and both +were confirmed against gufo's source rather than assumed: + +- **`general.file_type` is 15 vs 7.** gufo's `gguf_reader.cpp` states it is + *"deliberately not consulted… The Unsloth UD-\*_K_XL artifacts this repo + targets carry a stale one — UD-Q8_K_XL records ftype 15 (Q4_K_M) … The + tensor table always is."* The file gufo already served lies about itself + here, so gufo reads the tensor table. +- **Swift adds `qwen35.attention.recurrent_layers`.** gufo never reads that + key anywhere in `src/`; layer kinds come from `full_attention_interval`, + which both files carry as `4` and which agrees with it. + +### Serving shape + - **Context:** one session, full 262,144 tokens (the model's native window), `SESSIONS=1`. The fleet's single stream is unchanged. - **Spec method:** DFlash2, gufo's **adaptive** block policy - (`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the same - z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB). -- **Sampling:** greedy (`--temperature 0`), as every prior frank-large - served. gufo's per-model recommendation for Qwen3.8 thinking is *sampled* - (1.0 / 0.95 / top-k 20) — that is a separate decision, deliberately not - bundled into a runtime swap. + (`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the + **unchanged** z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB) — it did not + move with the target, which is why `DRAFTER_DIR` is a separate tunable + from `MODEL_DIR` and why the deployer's stray-run sweep keys on the + drafter's filename rather than the target's. +- **Sampling:** greedy (`--temperature 0`) server-side, as every prior + frank-large served. The harness overrides per request via `extraBody` + (1.0 / 0.95 / top-k 20). - **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture. ### Measured at the 2026-09-28 cutover -Live on Dankclaw after `deploy.sh dankclaw --service frank-large`: +Live on Dankclaw. The **target** numbers are the Swift-1.5 load test +(a standalone run under the production flags, service stopped first); the +**engine** numbers are from the gufo cutover earlier the same day and are +unchanged by the payload swap. -- **13 s** from `load_started` to `listening` (target 8 s, drafter 2 s, - session 3 s), against a 20-minute `LLM_WAIT` budget — a redeploy no longer - sits in a load wait. -- **RSS 30.6 GiB**, VSZ 87.3 GiB; host `used` 81 GiB of 121 GiB, i.e. the - same order as the llama.cpp deployment it replaced. -- A short-prompt live request returned - `Server-Timing: ttft;dur=549.8, inter_token;dur=24.2, max_inter_token;dur=166.9` - (≈41 tok/s). Short-prompt decode is gufo's *winning* regime; see the depth - table above before quoting it as the endpoint's number. -- Contract battery, all green: `/health` 200, `/ready` 200, - `/v1/models` → `frank-large` with `context_length 262144`, a chat - completion carrying both `content` and `reasoning_content`, a `tool_calls` - response with `finish_reason: tool_calls`, and SSE streaming. +- **9.2 s** `load_started` → `load_completed` for the Swift target, against a + 20-minute `LLM_WAIT` budget. The unsloth target loaded in 13 s. +- **66.5 GiB** of the 116.7 GiB iGPU device at context capacity 262,144; + host RSS 30.6 GiB. No loader warnings, no skipped sections. +- Contract battery, all green on the Swift target: `/v1/models` → + `context_length 262144`, a chat completion carrying both `content` and + `reasoning_content`, a `tool_calls` response with + `finish_reason: tool_calls`, SSE streaming, and a **two-turn tool loop** + (tool call → tool result → follow-up) with prior reasoning preserved. + Arithmetic spot-check: "17*23" → `391`. +- Behavioural spot-check that this is the abliterated model: asked for a + two-sentence lockpicking scene, it wrote one without hedging or refusing. +- From the gufo cutover, on the unsloth target: a short-prompt live request + returned + `Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (≈41 tok/s). + Short-prompt decode is gufo's *winning* regime; see the depth table above + before quoting it as the endpoint's number. + +## The template law: Qwen-Sharp is gone + +`scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is +wired into `frank-primary` as `R9V_CHAT_TEMPLATE_FILE` and it was +`frank-large`'s `--chat-template-file` under llama.cpp. **frank-large can no +longer use it**, and the reason is stronger than "gufo lacks the flag": + +> gufo has **no Jinja engine at all.** `src/models/qwen/chat_template.cpp` +> is 587 lines of hand-written C++ that appends strings to build ChatML. +> There is no `--chat-template-file`, no environment override, and no +> template body anywhere in the binary to point at. Qwen-Sharp is 453 lines +> of Jinja2 using `namespace()`, macros, `raise_exception` and image/video +> branches — supplying it would mean **reimplementing it in C++**, not +> handing gufo a file. + +Given the choice on 2026-09-28, the decision was to **ship on gufo's stock +Jinja and accept the loss**. What that gives up, concretely: + +| Qwen-Sharp feature | On gufo | +|---|---| +| `terse` system-prompt injection (the headline behaviour) | **gone** | +| in-band `<\|think_off\|>` / `<\|think_xhigh\|>` control | **gone** | +| `tool_call_format` `json` \| `xml` switch | **gone** | +| `max_tool_arg_chars` / `max_tool_response_chars` truncation | **gone** | +| `add_vision_id` → `Picture N:` prefixes | **gone** | +| effort instruction strings in the system prompt | kept (gufo has its own, same text) | +| `preserve_thinking` | kept (`--preserve-thinking on`) | + +The `terse` injection is the one that matters most here, and it is worth +being blunt about the combination: this endpoint now runs an **abliterated** +model *without* the prompt that tells it to stop waffling. If verbosity +shows up, the cheapest fix is to send the `terse` text as an ordinary +**system message from the client** (`models.yml` / the harness) rather than +through a template — that recovers the behaviour without forking the +renderer. The full-fidelity options are reverting to llama.cpp or porting +Qwen-Sharp into the C++ renderer; both are worse trades today, and the +second is the same class of change that produced the 2026-09-21 +"73,600-token prompt" incident. ## Client contract — what actually changed @@ -134,21 +226,22 @@ auto-discovers a `mmproj` beside the model; this payload has none). The differences worth knowing: -- **No custom chat template.** gufo compiles the official Qwen Jinja into the - binary and takes no `--chat-template-file`. The Qwen-Sharp v22.4.0 file - this endpoint used with llama.cpp is **no longer in the command line** — - the 2026-09-21 "73,600-token prompt" incident cannot recur, because there - is no fallback template to land in. Thinking defaults to on at `xhigh` - with prior reasoning preserved (`--think auto --preserve-thinking on`, - spelled out in the serve script). +- **No custom chat template — see + [The template law](#the-template-law-qwen-sharp-is-gone) above**, which is + the most consequential change on this endpoint. Thinking defaults to on at + `xhigh` with prior reasoning preserved (`--think auto + --preserve-thinking on`, spelled out in the serve script). - **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`, - `low`, `medium`, `high`, `xhigh`, `max`). Qwen-Sharp rejected anything - outside `{xhigh, medium, low}` with an HTTP 500, which is why `models.yml` - still carries a `reasoningEffortMap` fold for this model. **The fold is - kept on purpose**: removing it would silently move every `high` role from - `xhigh` to `high` effort in the same commit as the runtime swap. Drop it as - its own change once the effort curve is a decision rather than a side - effect. + `low`, `medium`, `high`, `xhigh`, `max`), all verified 200 against the + live endpoint. Qwen-Sharp rejected anything outside `{xhigh, medium, low}` + with an HTTP 500, which is why `models.yml` still carries a + `reasoningEffortMap` fold for this model. **The fold is kept on + purpose**: removing it would silently move every `high` role from `xhigh` + to `high` effort. Drop it as its own change. +- **Every field the harness sends is accepted.** `top_k`, `min_p`, + `presence_penalty`, `repetition_penalty` and `preserve_thinking` are all + outside gufo's documented request subset, and all returned 200. They are + applied from `models.yml`'s `extraBody` unchanged. - **No KV-cache quantization flag.** gufo picks its own KV dtype; there is no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it. Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in @@ -156,8 +249,8 @@ The differences worth knowing: - **Health/readiness split:** `GET /health` is process liveness and is what the deployer gates on; `GET /ready` is model/backend readiness. `GET /metrics` is Prometheus-format. -- **Long bodies:** `--max-request-bytes` defaults to 8 MiB and unknown - request fields are rejected rather than ignored. + +- **Long bodies:** `--max-request-bytes` defaults to 8 MiB. ## Failure modes worth knowing @@ -169,26 +262,44 @@ The differences worth knowing: `/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself (`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device time-slices instead of failing; four HIP processes is the device ceiling. -- **Load is seconds, not minutes** (8 s measured cold), so a redeploy no - longer spends 20 minutes in `LLM_WAIT`. +- **Load is seconds, not minutes** (9.2 s measured cold on the Swift + target), so a redeploy no longer spends 20 minutes in `LLM_WAIT`. +- **A payload is two files in two directories.** The target and the drafter + no longer live together, so "the payload is present" is not one check; the + deployer gates on both paths and the serve script refuses to start if + either is absent. A symlinked "one payload directory" would hide that. - **A v0.1.x runtime is a moving target.** Rebuilds are not automatic — `RUNTIME_DIR` names a version, and the deployer will not silently substitute another. ## Rollback -[[frank-large-unsloth-qwen38-27b-q8xl]] — the unsloth b11030 prebuilt is -still at `/home/dank/GitPackages/unsloth-llama-b11030/` and the payload is -the same directory the current runtime reads. Roll back by reverting -`RUNTIME_DIR` (and the gufo-only tunables) in `scripts/frank-large.env`, -restoring the serve script's llama.cpp command, and redeploying. +Two independent rollbacks, because two things changed on 2026-09-28: + +1. **Payload only** — repoint `MODEL_FILE`/`MODEL_DIR` in + `scripts/frank-large.env` at + `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf` + and redeploy. That directory is untouched on disk and the drafter already + lives in it. This is the fast way back to the stock Qwen weights on the + gufo runtime. +2. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]]: the unsloth b11030 + prebuilt is still at `/home/dank/GitPackages/unsloth-llama-b11030/`. + Revert `RUNTIME_DIR` and the gufo-only tunables, restore the serve + script's llama.cpp command, and redeploy. This is also the only way to + get `qwen-sharp-v22.4.0.jinja` back. # Citations - Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT) - Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo -- API contract: `docs/SERVER.md`; flags: `docs/CLI.md` +- API contract: `docs/SERVER.md`; flags: `docs/CLI.md`; the C++ renderer + that rules out a Jinja template: + `src/models/qwen/chat_template.cpp` (587 lines) +- GGUF header evidence for the payload swap: both files' headers were parsed + and diffed directly (architecture, all 866 tensor names in order, per-tensor + `ggml_type` and dims) before either was loaded - Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env` -- Payload: `scripts/download-frank-large.sh` +- Payload: `scripts/download-frank-large-swift.sh` (target), + `scripts/download-frank-large.sh` (DFlash2 drafter) - Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]] diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index dd384c3..f1a337f 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -2,6 +2,77 @@ ## 2026-09-28 +- **frank-large payload swapped to Swift-1.5 (uncensored)** (production + change, Dankclaw). Frank asked whether the UD Q8 of + `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` would load on + gufo, then said to do it. Port 7770, alias `frank-large`, one 262,144 + session, the DFlash2 drafter and the client contract are unchanged; the + 27B dense weights are an **abliterated (uncensored)** fine-tune of the + same base, under the **non-OSI SWIFT OPEN LICENSE 1.0**. + - **Compatibility was established from the GGUF headers, not from the + filename.** Both files were range-fetched and parsed: same + `general.architecture` (`qwen35`), same 866 tensors with **identical + names in identical order**, **zero** per-tensor `ggml_type` or dim + mismatches (Q8_0×453, F32×360, BF16×53), same MTP head at `blk.64`, same + geometry. The two scary metadata differences are both no-ops, confirmed + against gufo's source: `general.file_type` (15 vs 7) is *"deliberately + not consulted"* by `gguf_reader.cpp` — the file gufo already served + records 15 (Q4_K_M) while containing no Q4_K tensor — and Swift's extra + `qwen35.attention.recurrent_layers` key is **never read** by gufo + anywhere in `src/`; layer kinds come from `full_attention_interval`, + which both files carry as `4` and which agrees with it. + - **Fetched and verified:** 31,457,990,272 bytes, sha256 `9076f99a…` + matching the Hub LFS oid, via the new + `/scripts/download-frank-large-swift.sh`. It sends **no auth header** — + the repo is public and ungated, and a token on the curl command line is + world-readable via `/proc//cmdline`. The anonymous transfer ran at + ~1.8 MiB/s; resuming with the host's token through a **mode-600 + `curl -K` config** took it to ~7 MiB/s. The download survived an ssh + drop by running `setsid nohup`, not under the session. + - **Load test, service stopped first** (the GPU policy): **9.2 s** from + `load_started` to `load_completed`, 66.5 GiB of the 116.7 GiB iGPU at + context capacity 262,144, no loader warnings. Contract battery green: + `context_length 262144`, `content` + `reasoning_content`, + `finish_reason: tool_calls`, SSE, and a **two-turn tool loop** with prior + reasoning preserved. Spot-checks: "17*23" → `391`; asked for a + lockpicking scene it wrote one without hedging, confirming the + abliteration. + - **THE TEMPLATE LAW IS GONE, by decision.** Frank asked to "use our chat + template"; gufo **cannot** take it. This is not a missing flag — + `src/models/qwen/chat_template.cpp` is 587 lines of hand-written C++ + that appends strings, with no Jinja engine, no `--chat-template-file`, + no env override and no template body in the binary. Qwen-Sharp + v22.4.0 is 453 lines of Jinja2 with `namespace()`, macros and + `raise_exception`, so supplying it is a **port, not a config line**. + Frank chose to ship on gufo's stock renderer and accept the loss: the + `terse` system-prompt injection, in-band `<|think_*|>` control, the + `json|xml` `tool_call_format` switch, tool-argument truncation and + `add_vision_id` prefixes are all gone. gufo's renderer does keep the + effort instruction strings and `preserve_thinking`. **Worth watching: + an abliterated model now runs without the prompt that tells it to stop + waffling** — the cheap fix is to send the `terse` text as an ordinary + system message from `models.yml`, not to touch a template. + - **The payload is now two files in two directories.** The target moved + repo; the DFlash2 drafter did not move, so `DRAFTER_DIR` is its own + tunable. That broke the deployer's stray-run sweep, which keyed on the + *target* filename and would have silently stopped matching — a stray + run would then have held 7770 past the port gate. It now keys on the + drafter, which every frank-large command line carries via + `--dflash-model` and which did not move. + - **`/tests/frank_large_unit.bats`** — 20 assertions. The mutation pass + found **four real gaps** on the first run, all now fixed: two fetch + assertions were satisfied by a **header comment** rather than the URL + assignment, so repointing a repo left them green; and nothing bound the + env's `MODEL_FILE`/`DRAFTER_FILE` to the filenames the fetch scripts + actually download, so a re-upload or an env edit would break the + endpoint with a "missing asset" no test predicted. There is now an + explicit cross-file invariant. A second harness bug worth recording: + a `bats --filter` that matches no test reports zero `not ok`, which + looks exactly like a mutation that survived — the harness now asserts + the `1..N` count before believing a result. 16/16 mutations caught. + - Docs: runbook gained a **Payload** section (the header evidence), + **The template law** section and a two-way **Rollback**; AGENTS.md + fleet table, frank-large section and rollback. - **frank-large runtime swapped to gufo** (production change, Dankclaw). Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, diff --git a/tests/frank_large_unit.bats b/tests/frank_large_unit.bats index 0b6be28..c430003 100755 --- a/tests/frank_large_unit.bats +++ b/tests/frank_large_unit.bats @@ -2,10 +2,12 @@ # Seam: the frank-large root unit on dankclaw. # # Since 2026-09-28 this endpoint is served by gufo (gufo-org/gufo), the -# Strix Halo inference engine, built from the pinned v0.1.1 tag and serving -# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter. It replaced -# the stock llama.cpp prebuilt (unsloth b11030) on the same port, the same -# alias and the same weights. +# Strix Halo inference engine built from the pinned v0.1.1 tag, serving +# Swift-1.5-Qwen3.8-27B-Uncensored (abliterated) UD-Q8_K_XL with the z-lab +# DFlash2 Q4_K_M drafter. Two changes landed on this endpoint that day: the +# engine (from the stock llama.cpp prebuilt) and the payload (from +# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL). Port, alias and client contract did +# not move. # # Most of this seam runs the serve script against a stub engine and asserts # on the argv it actually execs, so what is pinned is the command the @@ -17,15 +19,19 @@ load helpers ENV_FILE="$REPO_ROOT/scripts/frank-large.env" UNIT="$REPO_ROOT/scripts/frank-large.service" SERVE="$REPO_ROOT/scripts/serve-frank-large.sh" -FETCH="$REPO_ROOT/scripts/download-frank-large.sh" +FETCH="$REPO_ROOT/scripts/download-frank-large-swift.sh" +DRAFTER_FETCH="$REPO_ROOT/scripts/download-frank-large.sh" BUILD="$REPO_ROOT/scripts/build-gufo.sh" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" - +# The deployed target's filename, kept here so a rename in the env file and a +# rename in this stub cannot drift apart unnoticed. +TARGET_STUB="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf" setup() { STAGE="$(mktemp -d)" RUNTIME_DIR="$STAGE/gufo" MODEL_DIR="$STAGE/models" - mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR" + DRAFTER_DIR="$STAGE/draft" + mkdir -p "$RUNTIME_DIR/bin" "$MODEL_DIR" "$DRAFTER_DIR" # A stub engine: the serve script execs it, so $@ is exactly what gufo # would be invoked with. cat > "$RUNTIME_DIR/bin/gufo" <<'STUB' @@ -33,12 +39,17 @@ setup() { printf '%s\n' "$@" STUB chmod +x "$RUNTIME_DIR/bin/gufo" - : > "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf" - : > "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" + # The target and the drafter live in SEPARATE directories since the + # 2026-09-28 payload swap: the target came from a different repository, + # the DFlash2 drafter did not move. A stub that colocated them would + # never notice DRAFTER_DIR going stale. + : > "$MODEL_DIR/$TARGET_STUB" + : > "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" STUB_ENV="$STAGE/stub.env" cat > "$STUB_ENV" < "$STAGE/shipped.env" serve_argv "$STAGE/shipped.env" [ "$status" -eq 0 ] @@ -132,6 +149,10 @@ argv_value() { [ "$(argv_value --speculative)" = "dflash2" ] [ "$(argv_value --draft-tokens)" -gt 0 ] [ "$(argv_value --draft-policy)" = "adaptive" ] + # The shipped file must name the target the fleet actually serves, not a + # path that merely happens to exist. + [ "$(argv_value --model)" = "$MODEL_DIR/$TARGET_STUB" ] + [ "$(argv_value --dflash-model)" = "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" ] } @test "the engine is pointed at the payload that is actually on disk" { @@ -173,30 +194,68 @@ argv_value() { ! argv_has serve } -@test "the serve script refuses a missing drafter with the fetch pointer" { - rm -f "$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" +@test "the serve script refuses a missing drafter with a fetch pointer" { + # The drafter lives in its own directory now, so the target can be present + # and this still has to be caught. + rm -f "$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf" serve_argv [ "$status" -ne 0 ] [[ "$output" == *"missing asset"* ]] - [[ "$output" == *"download-frank-large.sh"* ]] + [[ "$output" == *"Qwen3.8-27B-DFlash2-Q4_K_M.gguf"* ]] + [[ "$output" == *"download-frank-large"* ]] ! argv_has serve } @test "the serve script refuses a missing target and names the file" { - rm -f "$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf" + rm -f "$MODEL_DIR/$TARGET_STUB" serve_argv [ "$status" -ne 0 ] [[ "$output" == *"missing asset"* ]] - [[ "$output" == *"Qwen3.8-27B-UD-Q8_K_XL.gguf"* ]] + [[ "$output" == *"$TARGET_STUB"* ]] } -@test "the payload is fetched resumably and verified against declared sizes + sha256" { - grep -q 'unsloth/Qwen3.8-27B-GGUF' "$FETCH" - grep -q 'z-lab/Qwen3.8-27B-DFlash2-GGUF' "$FETCH" - grep -q -- '-fL -C -' "$FETCH" - # Each file's local sha256 is compared to its declared oid and a - # mismatch fails the run. - grep -q 'SHA256 MISMATCH' "$FETCH" +@test "both payload files are fetched resumably and verified against declared sizes + sha256" { + # The target and the drafter come from DIFFERENT repositories, fetched by + # two scripts. Anchor the repo checks to the URL ASSIGNMENT, not to the + # file: the same repo string also appears in each script's header comment, + # so a plain grep stayed green after the URL was repointed at another + # repository -- the script would then download something the serve script + # never asked for. + grep -qE '^URL="https://huggingface\.co/ajgazin/Swift-1\.5-Qwen3\.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/' "$FETCH" + grep -qE 'https://huggingface\.co/z-lab/Qwen3\.8-27B-DFlash2-GGUF/resolve/main/' "$DRAFTER_FETCH" + for f in "$FETCH" "$DRAFTER_FETCH"; do + # -f so an HTTP error is a failure rather than a small "artifact"; + # -C - so an interrupted 29 GiB run resumes. + grep -q -- '-fL -C -' "$f" + # The size and the sha256 are compared after the transfer, not trusted. + grep -q 'SHA256 MISMATCH' "$f" + grep -q 'SIZE MISMATCH' "$f" + done +} + +@test "the serve script and the fetch scripts agree on the payload filenames" { + # The serve script's MODEL_FILE and the fetch script's REL are two + # independent spellings of one filename, and a repo is not required to + # keep them in step: a re-upload under a new name, or an env edited + # without the fetch, leaves the endpoint unable to start with a + # "missing asset" that no test would have predicted. Bind them here. + local served fetched drafter_served drafter_fetched + served="$(basename "$(grep '^MODEL_FILE=' "$ENV_FILE" | cut -d= -f2-)")" + fetched="$(sed -n 's/^REL="\(.*\)"$/\1/p' "$FETCH")" + [ -n "$served" ] + [ "$served" = "$fetched" ] + drafter_served="$(basename "$(grep '^DRAFTER_FILE=' "$ENV_FILE" | cut -d= -f2-)")" + drafter_fetched="$(sed -n 's/^ *"\(Qwen3\.8-27B-DFlash2[^ ]*\.gguf\) .*/\1/p' "$DRAFTER_FETCH")" + [ -n "$drafter_served" ] + [ "$drafter_served" = "$drafter_fetched" ] +} + +@test "the drafter is still fetched by the script that carries its Hub repo" { + # Guards a future "simplification" that folds the drafter into the Swift + # script and silently orphans the artifact the serve script still points + # at via DRAFTER_DIR. + grep -q 'Qwen3.8-27B-UD-Q8_K_XL' "$DRAFTER_FETCH" + ! grep -q 'DFlash2' "$FETCH" } @test "the build script refuses a source tree that is not the pinned commit" { @@ -247,11 +306,15 @@ argv_value() { [ "$bail" -lt "$stop" ] } -@test "the stray-run sweep cannot take frank-small down" { - # frank-small on this host serves a different GGUF. The sweep must match - # the model path, not a binary name, or a frank-large deploy restarts the - # 3090 endpoint. - grep -q "pkill -f 'Qwen3.8-27B-UD-Q8_K_XL'" "$REMOTE_DEPLOY" +@test "the stray-run sweep keys on the drafter and cannot take frank-small down" { + # The sweep used to key on the TARGET filename, which the 2026-09-28 + # payload swap moved to a different repository. It would have stopped + # matching the live process, and a stray run would then hold 7770 past the + # port gate and fail the deploy. The drafter is the part that did NOT + # move, and every frank-large command line carries it via --dflash-model. + grep -q "pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M'" "$REMOTE_DEPLOY" + # frank-small serves a different GGUF on the 3090 with no DFlash2 drafter, + # so a binary-name sweep is the only thing that would endanger it. local large large="$(sed -n '/^deploy_llm_large() {/,/^}/p' "$REMOTE_DEPLOY")" ! grep -q 'pkill -x llama-server' <<<"$large"