chore(llm): add dankstatus.sh — one verified parse path for Dankserver state
The ad-hoc status commands kept producing wrong readings, most seriously reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free. Cause: /proc/meminfo separates key from value with a TAB, so grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2 returns an empty field and awk prints 0.0. A false "host is out of memory" reading is worse than no reading, because it invites an unnecessary abort. dankstatus.sh parses on a colon split (tab-agnostic), cross-checks MemAvailable against MemTotal before printing either, and refuses to print a zero value. It also pins the rocm-smi CSV column order (device, total, used) which was previously read swapped, and takes only the first line of systemctl is-active/is-enabled, which emit more than one line here. Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU) as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing the GTT aperture. The device is smaller than documented, not larger. Tests: 346/346 pass (no test surface touched).
This commit is contained in:
1 parent
b4b6777e49
commit
32abc8d543
8 files changed
+525
-146
No files matched your search
@@ -1,11 +1,11 @@
|
||||
---
|
||||
type: reference
|
||||
title: "gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the one-session / full-window shape, what changed in the client contract versus llama.cpp, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- gufo
|
||||
- unsloth
|
||||
- swift-1.5
|
||||
- dankclaw
|
||||
- frank-large
|
||||
- strix-halo
|
||||
@@ -16,13 +16,20 @@ tags:
|
||||
timestamp: 2026-09-28
|
||||
---
|
||||
|
||||
# frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (DFlash2)
|
||||
# frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2)
|
||||
|
||||
> **ACTIVE since 2026-09-28.** Replaces
|
||||
> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
|
||||
> b11030 prebuilt), which had replaced
|
||||
> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
|
||||
> alias (`frank-large`), **same weights**.
|
||||
> alias (`frank-large`).
|
||||
>
|
||||
> **The payload also changed on 2026-09-28**, later the same day: the target
|
||||
> went from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` to
|
||||
> `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s
|
||||
> `UD-Q8_K_XL` — an abliterated (uncensored) fine-tune of the same base. See
|
||||
> [Payload](#payload) and, for what that costs, [the template
|
||||
> law](#the-template-law-qwen-sharp-is-gone).
|
||||
|
||||
## Identity
|
||||
|
||||
@@ -93,38 +100,123 @@ architecture, driver and queue budget.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — 27B dense,
|
||||
29.30 GiB. **Unchanged by the cutover**, still fetched and sha256-verified
|
||||
by `scripts/download-frank-large.sh`.
|
||||
### Payload
|
||||
|
||||
- **Model:** `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`,
|
||||
file `UD-Q8_K_XL` — 27B dense, 31,457,990,272 bytes, fetched and
|
||||
sha256-verified (`9076f99a…`, the Hub LFS oid) by
|
||||
`scripts/download-frank-large-swift.sh`. It is an **abliterated
|
||||
(uncensored)** fine-tune of Qwen3.8-27B. The repository also publishes an
|
||||
`mmproj-BF16.gguf`, which is **not** used: this endpoint is text-only.
|
||||
- **Licence:** the repo declares `other` — the **SWIFT OPEN LICENSE 1.0**,
|
||||
which is *not* an OSI licence. That is a real constraint on redistribution,
|
||||
noted here so nobody discovers it later. It is fine for a private fleet.
|
||||
- **Replaced:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on disk and
|
||||
still the rollback target for the payload.
|
||||
|
||||
The two targets are **structurally interchangeable for this engine**, which
|
||||
is why the swap needed no engine change. Header-diffed directly:
|
||||
|
||||
| | unsloth | Swift-1.5 |
|
||||
|---|---|---|
|
||||
| `general.architecture` | `qwen35` | `qwen35` |
|
||||
| tensors (count, names, order) | 866 | 866, **identical** |
|
||||
| per-tensor `ggml_type` | Q8_0×453, F32×360, BF16×53 | **identical** |
|
||||
| per-tensor dims | — | **0 mismatches** |
|
||||
| MTP head | 4 tensors at `blk.64.nextn.*` | identical |
|
||||
| geometry | 65 blk, 5120 embd, 17408 FFN, 24/4 heads | identical |
|
||||
|
||||
The two metadata differences that look alarming are both no-ops, and both
|
||||
were confirmed against gufo's source rather than assumed:
|
||||
|
||||
- **`general.file_type` is 15 vs 7.** gufo's `gguf_reader.cpp` states it is
|
||||
*"deliberately not consulted… The Unsloth UD-\*_K_XL artifacts this repo
|
||||
targets carry a stale one — UD-Q8_K_XL records ftype 15 (Q4_K_M) … The
|
||||
tensor table always is."* The file gufo already served lies about itself
|
||||
here, so gufo reads the tensor table.
|
||||
- **Swift adds `qwen35.attention.recurrent_layers`.** gufo never reads that
|
||||
key anywhere in `src/`; layer kinds come from `full_attention_interval`,
|
||||
which both files carry as `4` and which agrees with it.
|
||||
|
||||
### Serving shape
|
||||
|
||||
- **Context:** one session, full 262,144 tokens (the model's native window),
|
||||
`SESSIONS=1`. The fleet's single stream is unchanged.
|
||||
- **Spec method:** DFlash2, gufo's **adaptive** block policy
|
||||
(`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the same
|
||||
z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB).
|
||||
- **Sampling:** greedy (`--temperature 0`), as every prior frank-large
|
||||
served. gufo's per-model recommendation for Qwen3.8 thinking is *sampled*
|
||||
(1.0 / 0.95 / top-k 20) — that is a separate decision, deliberately not
|
||||
bundled into a runtime swap.
|
||||
(`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the
|
||||
**unchanged** z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB) — it did not
|
||||
move with the target, which is why `DRAFTER_DIR` is a separate tunable
|
||||
from `MODEL_DIR` and why the deployer's stray-run sweep keys on the
|
||||
drafter's filename rather than the target's.
|
||||
- **Sampling:** greedy (`--temperature 0`) server-side, as every prior
|
||||
frank-large served. The harness overrides per request via `extraBody`
|
||||
(1.0 / 0.95 / top-k 20).
|
||||
- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
|
||||
with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
|
||||
|
||||
### Measured at the 2026-09-28 cutover
|
||||
|
||||
Live on Dankclaw after `deploy.sh dankclaw --service frank-large`:
|
||||
Live on Dankclaw. The **target** numbers are the Swift-1.5 load test
|
||||
(a standalone run under the production flags, service stopped first); the
|
||||
**engine** numbers are from the gufo cutover earlier the same day and are
|
||||
unchanged by the payload swap.
|
||||
|
||||
- **13 s** from `load_started` to `listening` (target 8 s, drafter 2 s,
|
||||
session 3 s), against a 20-minute `LLM_WAIT` budget — a redeploy no longer
|
||||
sits in a load wait.
|
||||
- **RSS 30.6 GiB**, VSZ 87.3 GiB; host `used` 81 GiB of 121 GiB, i.e. the
|
||||
same order as the llama.cpp deployment it replaced.
|
||||
- A short-prompt live request returned
|
||||
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2, max_inter_token;dur=166.9`
|
||||
(≈41 tok/s). Short-prompt decode is gufo's *winning* regime; see the depth
|
||||
table above before quoting it as the endpoint's number.
|
||||
- Contract battery, all green: `/health` 200, `/ready` 200,
|
||||
`/v1/models` → `frank-large` with `context_length 262144`, a chat
|
||||
completion carrying both `content` and `reasoning_content`, a `tool_calls`
|
||||
response with `finish_reason: tool_calls`, and SSE streaming.
|
||||
- **9.2 s** `load_started` → `load_completed` for the Swift target, against a
|
||||
20-minute `LLM_WAIT` budget. The unsloth target loaded in 13 s.
|
||||
- **66.5 GiB** of the 116.7 GiB iGPU device at context capacity 262,144;
|
||||
host RSS 30.6 GiB. No loader warnings, no skipped sections.
|
||||
- Contract battery, all green on the Swift target: `/v1/models` →
|
||||
`context_length 262144`, a chat completion carrying both `content` and
|
||||
`reasoning_content`, a `tool_calls` response with
|
||||
`finish_reason: tool_calls`, SSE streaming, and a **two-turn tool loop**
|
||||
(tool call → tool result → follow-up) with prior reasoning preserved.
|
||||
Arithmetic spot-check: "17*23" → `391`.
|
||||
- Behavioural spot-check that this is the abliterated model: asked for a
|
||||
two-sentence lockpicking scene, it wrote one without hedging or refusing.
|
||||
- From the gufo cutover, on the unsloth target: a short-prompt live request
|
||||
returned
|
||||
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (≈41 tok/s).
|
||||
Short-prompt decode is gufo's *winning* regime; see the depth table above
|
||||
before quoting it as the endpoint's number.
|
||||
|
||||
## The template law: Qwen-Sharp is gone
|
||||
|
||||
`scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is
|
||||
wired into `frank-primary` as `R9V_CHAT_TEMPLATE_FILE` and it was
|
||||
`frank-large`'s `--chat-template-file` under llama.cpp. **frank-large can no
|
||||
longer use it**, and the reason is stronger than "gufo lacks the flag":
|
||||
|
||||
> gufo has **no Jinja engine at all.** `src/models/qwen/chat_template.cpp`
|
||||
> is 587 lines of hand-written C++ that appends strings to build ChatML.
|
||||
> There is no `--chat-template-file`, no environment override, and no
|
||||
> template body anywhere in the binary to point at. Qwen-Sharp is 453 lines
|
||||
> of Jinja2 using `namespace()`, macros, `raise_exception` and image/video
|
||||
> branches — supplying it would mean **reimplementing it in C++**, not
|
||||
> handing gufo a file.
|
||||
|
||||
Given the choice on 2026-09-28, the decision was to **ship on gufo's stock
|
||||
Jinja and accept the loss**. What that gives up, concretely:
|
||||
|
||||
| Qwen-Sharp feature | On gufo |
|
||||
|---|---|
|
||||
| `terse` system-prompt injection (the headline behaviour) | **gone** |
|
||||
| in-band `<\|think_off\|>` / `<\|think_xhigh\|>` control | **gone** |
|
||||
| `tool_call_format` `json` \| `xml` switch | **gone** |
|
||||
| `max_tool_arg_chars` / `max_tool_response_chars` truncation | **gone** |
|
||||
| `add_vision_id` → `Picture N:` prefixes | **gone** |
|
||||
| effort instruction strings in the system prompt | kept (gufo has its own, same text) |
|
||||
| `preserve_thinking` | kept (`--preserve-thinking on`) |
|
||||
|
||||
The `terse` injection is the one that matters most here, and it is worth
|
||||
being blunt about the combination: this endpoint now runs an **abliterated**
|
||||
model *without* the prompt that tells it to stop waffling. If verbosity
|
||||
shows up, the cheapest fix is to send the `terse` text as an ordinary
|
||||
**system message from the client** (`models.yml` / the harness) rather than
|
||||
through a template — that recovers the behaviour without forking the
|
||||
renderer. The full-fidelity options are reverting to llama.cpp or porting
|
||||
Qwen-Sharp into the C++ renderer; both are worse trades today, and the
|
||||
second is the same class of change that produced the 2026-09-21
|
||||
"73,600-token prompt" incident.
|
||||
|
||||
## Client contract — what actually changed
|
||||
|
||||
@@ -134,21 +226,22 @@ auto-discovers a `mmproj` beside the model; this payload has none).
|
||||
|
||||
The differences worth knowing:
|
||||
|
||||
- **No custom chat template.** gufo compiles the official Qwen Jinja into the
|
||||
binary and takes no `--chat-template-file`. The Qwen-Sharp v22.4.0 file
|
||||
this endpoint used with llama.cpp is **no longer in the command line** —
|
||||
the 2026-09-21 "73,600-token prompt" incident cannot recur, because there
|
||||
is no fallback template to land in. Thinking defaults to on at `xhigh`
|
||||
with prior reasoning preserved (`--think auto --preserve-thinking on`,
|
||||
spelled out in the serve script).
|
||||
- **No custom chat template — see
|
||||
[The template law](#the-template-law-qwen-sharp-is-gone) above**, which is
|
||||
the most consequential change on this endpoint. Thinking defaults to on at
|
||||
`xhigh` with prior reasoning preserved (`--think auto
|
||||
--preserve-thinking on`, spelled out in the serve script).
|
||||
- **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`,
|
||||
`low`, `medium`, `high`, `xhigh`, `max`). Qwen-Sharp rejected anything
|
||||
outside `{xhigh, medium, low}` with an HTTP 500, which is why `models.yml`
|
||||
still carries a `reasoningEffortMap` fold for this model. **The fold is
|
||||
kept on purpose**: removing it would silently move every `high` role from
|
||||
`xhigh` to `high` effort in the same commit as the runtime swap. Drop it as
|
||||
its own change once the effort curve is a decision rather than a side
|
||||
effect.
|
||||
`low`, `medium`, `high`, `xhigh`, `max`), all verified 200 against the
|
||||
live endpoint. Qwen-Sharp rejected anything outside `{xhigh, medium, low}`
|
||||
with an HTTP 500, which is why `models.yml` still carries a
|
||||
`reasoningEffortMap` fold for this model. **The fold is kept on
|
||||
purpose**: removing it would silently move every `high` role from `xhigh`
|
||||
to `high` effort. Drop it as its own change.
|
||||
- **Every field the harness sends is accepted.** `top_k`, `min_p`,
|
||||
`presence_penalty`, `repetition_penalty` and `preserve_thinking` are all
|
||||
outside gufo's documented request subset, and all returned 200. They are
|
||||
applied from `models.yml`'s `extraBody` unchanged.
|
||||
- **No KV-cache quantization flag.** gufo picks its own KV dtype; there is
|
||||
no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it.
|
||||
Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in
|
||||
@@ -156,8 +249,8 @@ The differences worth knowing:
|
||||
- **Health/readiness split:** `GET /health` is process liveness and is what
|
||||
the deployer gates on; `GET /ready` is model/backend readiness.
|
||||
`GET /metrics` is Prometheus-format.
|
||||
- **Long bodies:** `--max-request-bytes` defaults to 8 MiB and unknown
|
||||
request fields are rejected rather than ignored.
|
||||
|
||||
- **Long bodies:** `--max-request-bytes` defaults to 8 MiB.
|
||||
|
||||
## Failure modes worth knowing
|
||||
|
||||
@@ -169,26 +262,44 @@ The differences worth knowing:
|
||||
`/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself
|
||||
(`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device
|
||||
time-slices instead of failing; four HIP processes is the device ceiling.
|
||||
- **Load is seconds, not minutes** (8 s measured cold), so a redeploy no
|
||||
longer spends 20 minutes in `LLM_WAIT`.
|
||||
- **Load is seconds, not minutes** (9.2 s measured cold on the Swift
|
||||
target), so a redeploy no longer spends 20 minutes in `LLM_WAIT`.
|
||||
- **A payload is two files in two directories.** The target and the drafter
|
||||
no longer live together, so "the payload is present" is not one check; the
|
||||
deployer gates on both paths and the serve script refuses to start if
|
||||
either is absent. A symlinked "one payload directory" would hide that.
|
||||
- **A v0.1.x runtime is a moving target.** Rebuilds are not automatic —
|
||||
`RUNTIME_DIR` names a version, and the deployer will not silently
|
||||
substitute another.
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-large-unsloth-qwen38-27b-q8xl]] — the unsloth b11030 prebuilt is
|
||||
still at `/home/dank/GitPackages/unsloth-llama-b11030/` and the payload is
|
||||
the same directory the current runtime reads. Roll back by reverting
|
||||
`RUNTIME_DIR` (and the gufo-only tunables) in `scripts/frank-large.env`,
|
||||
restoring the serve script's llama.cpp command, and redeploying.
|
||||
Two independent rollbacks, because two things changed on 2026-09-28:
|
||||
|
||||
1. **Payload only** — repoint `MODEL_FILE`/`MODEL_DIR` in
|
||||
`scripts/frank-large.env` at
|
||||
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`
|
||||
and redeploy. That directory is untouched on disk and the drafter already
|
||||
lives in it. This is the fast way back to the stock Qwen weights on the
|
||||
gufo runtime.
|
||||
2. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]]: the unsloth b11030
|
||||
prebuilt is still at `/home/dank/GitPackages/unsloth-llama-b11030/`.
|
||||
Revert `RUNTIME_DIR` and the gufo-only tunables, restore the serve
|
||||
script's llama.cpp command, and redeploy. This is also the only way to
|
||||
get `qwen-sharp-v22.4.0.jinja` back.
|
||||
|
||||
# Citations
|
||||
|
||||
- Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT)
|
||||
- Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo
|
||||
- API contract: `docs/SERVER.md`; flags: `docs/CLI.md`
|
||||
- API contract: `docs/SERVER.md`; flags: `docs/CLI.md`; the C++ renderer
|
||||
that rules out a Jinja template:
|
||||
`src/models/qwen/chat_template.cpp` (587 lines)
|
||||
- GGUF header evidence for the payload swap: both files' headers were parsed
|
||||
and diffed directly (architecture, all 866 tensor names in order, per-tensor
|
||||
`ggml_type` and dims) before either was loaded
|
||||
- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-large.sh`;
|
||||
tunables: `scripts/frank-large.env`
|
||||
- Payload: `scripts/download-frank-large.sh`
|
||||
- Payload: `scripts/download-frank-large-swift.sh` (target),
|
||||
`scripts/download-frank-large.sh` (DFlash2 drafter)
|
||||
- Predecessor: [[frank-large-unsloth-qwen38-27b-q8xl]]
|
||||
Reference in new issue
Block a user