chore(llm): add dankstatus.sh — one verified parse path for Dankserver state
The ad-hoc status commands kept producing wrong readings, most seriously reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free. Cause: /proc/meminfo separates key from value with a TAB, so grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2 returns an empty field and awk prints 0.0. A false "host is out of memory" reading is worse than no reading, because it invites an unnecessary abort. dankstatus.sh parses on a colon split (tab-agnostic), cross-checks MemAvailable against MemTotal before printing either, and refuses to print a zero value. It also pins the rocm-smi CSV column order (device, total, used) which was previously read swapped, and takes only the first line of systemctl is-active/is-enabled, which emit more than one line here. Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU) as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing the GTT aperture. The device is smaller than documented, not larger. Tests: 346/346 pass (no test surface touched).
This commit is contained in:
1 parent
b4b6777e49
commit
32abc8d543
8 files changed
+525
-146
No files matched your search
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 29.3 GiB) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, compiled-in Qwen Jinja, text only | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
@@ -277,20 +277,30 @@ FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
|
||||
projector's audio encoder (frank-small transcribes WAV verbatim even though
|
||||
`audio` is omitted from its `input` list — OMP's schema rejects it).
|
||||
|
||||
### frank-large — gufo Qwen3.8-27B UD-Q8_K_XL (Dankclaw, ACTIVE)
|
||||
### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)
|
||||
|
||||
**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a
|
||||
vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 /
|
||||
Radeon 8060S, `gfx1151`) — as a **root systemd unit**
|
||||
(`scripts/frank-large.service`, `User=dank`), serving
|
||||
`unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` (27B dense, 29.3 GiB,
|
||||
sha256-verified) on port 7770 from the gfx1151 iGPU. One session, full
|
||||
262,144 ctx (the model's native window, `SESSIONS=1`), DFlash2 spec decode
|
||||
(the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's **adaptive** block
|
||||
policy, `--draft-tokens 7`), greedy sampling, reasoning extracted to
|
||||
`reasoning_content`, and **no vision projector**. It replaced the unsloth
|
||||
llama.cpp b11030 prebuilt, which had replaced the Kairic Edge ROCmFPX
|
||||
build. Cut over 2026-09-28.
|
||||
`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s `UD-Q8_K_XL`
|
||||
(27B dense, 29.3 GiB, sha256-verified; **abliterated/uncensored**, and under
|
||||
the *non-OSI* SWIFT OPEN LICENSE 1.0) on port 7770 from the gfx1151 iGPU.
|
||||
One session, full 262,144 ctx (the model's native window, `SESSIONS=1`),
|
||||
DFlash2 spec decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's
|
||||
**adaptive** block policy, `--draft-tokens 7`), greedy sampling, reasoning
|
||||
extracted to `reasoning_content`, and **no vision projector**.
|
||||
|
||||
**Two swaps landed on 2026-09-28**, both same day: the engine (from the
|
||||
unsloth llama.cpp b11030 prebuilt, which had replaced Kairic Edge ROCmFPX)
|
||||
and then the payload (from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on
|
||||
disk and the payload-only rollback). The two targets are structurally
|
||||
interchangeable for gufo — same `qwen35` architecture, the same 866 tensors
|
||||
in the same order, the same per-tensor Q8_0/F32/BF16 layout, the same MTP
|
||||
head at `blk.64` — verified by parsing and diffing both GGUF headers before
|
||||
either was loaded. **The DFlash2 drafter did not move**, so it has its own
|
||||
`DRAFTER_DIR` and the deployer's stray-run sweep keys on the drafter's
|
||||
filename, not the target's.
|
||||
|
||||
**The runtime is built from source on the host** —
|
||||
`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with
|
||||
@@ -316,15 +326,27 @@ It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for
|
||||
matters because ComfyUI holds queues on the same device; four HIP processes
|
||||
is the device ceiling.
|
||||
|
||||
**The client contract changed in three ways worth knowing.** There is no
|
||||
`--chat-template-file`: the official Qwen Jinja is compiled in, so the
|
||||
Qwen-Sharp 73k-token-prompt failure mode cannot recur. `reasoning_effort`
|
||||
now accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap`
|
||||
fold for frank-large is no longer *required* — it is kept deliberately, so
|
||||
that removing it does not silently move every `high` role from `xhigh` to
|
||||
`high` effort inside a runtime swap. And there is no KV-quantization or
|
||||
prompt-cache-RAM flag; prompt reuse is on by default and `--cache-disk`
|
||||
is opt-in and off.
|
||||
**The client contract changed, and the chat template is the sharp edge.**
|
||||
**gufo has no Jinja engine at all** — `src/models/qwen/chat_template.cpp` is
|
||||
587 lines of hand-written C++ that appends strings, with no
|
||||
`--chat-template-file`, no env override, and no template body to point at.
|
||||
`scripts/qwen-sharp-v22.4.0.jinja` therefore **cannot** be supplied to
|
||||
frank-large: that is a port, not a config line. Given the choice on
|
||||
2026-09-28 the decision was to ship on gufo's stock renderer and accept the
|
||||
loss — the `terse` system-prompt injection, the in-band `<|think_*|>`
|
||||
control, the `json|xml` `tool_call_format` switch, the tool-argument
|
||||
truncation and `add_vision_id` prefixes are all gone. gufo's own renderer
|
||||
does keep the effort instruction strings and `preserve_thinking`, which are
|
||||
what the harness depends on. **This is the thing to watch**: the endpoint
|
||||
now runs an *abliterated* model without the prompt that tells it to stop
|
||||
waffling. The cheap fix, if verbosity shows up, is to send the `terse` text
|
||||
as an ordinary system message from `models.yml` — not a template change.
|
||||
`reasoning_effort` accepts the full OpenAI set (all six levels verified
|
||||
200), so `models.yml`'s `reasoningEffortMap` fold is no longer *required* —
|
||||
it is kept deliberately, so that removing it does not silently move every
|
||||
`high` role from `xhigh` to `high` effort inside a swap. And there is no
|
||||
KV-quantization or prompt-cache-RAM flag; prompt reuse is on by default and
|
||||
`--cache-disk` is opt-in and off.
|
||||
|
||||
**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables
|
||||
(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5×
|
||||
@@ -336,20 +358,25 @@ The Q4_K_XL target shows no such regression; revisiting the quantization is
|
||||
a separate decision from the runtime. Peak memory is 50.58 GiB at the full
|
||||
262,144 context against the iGPU's 114 GiB GTT aperture.
|
||||
|
||||
The serve script (`scripts/serve-frank-large.sh`) validates binary → main
|
||||
The serve script (`scripts/serve-frank-large.sh`) validates runtime → target
|
||||
GGUF → drafter and exits with a pointer message if any is missing, then
|
||||
`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model
|
||||
…--served-model-name frank-large --context 262144 --speculative dflash2 …`.
|
||||
The deployer's `deploy_llm_large` gates on the **runtime binary** as well
|
||||
as the payload, and returns before it stops anything.
|
||||
as both payload paths, and returns before it stops anything.
|
||||
|
||||
Fetch: `scripts/download-frank-large.sh` (per-file, each file verified
|
||||
against its declared byte count + sha256; `--verify-only` re-checks). The
|
||||
payload is unchanged by every frank-large runtime swap since 2026-09-21.
|
||||
Fetch: `scripts/download-frank-large-swift.sh` (the target; per-file byte
|
||||
count + sha256 against the Hub LFS oid, no auth header — the repo is public
|
||||
and a token on the curl command line is world-readable) and
|
||||
`scripts/download-frank-large.sh` (the DFlash2 drafter, which did not move).
|
||||
Both take `--verify-only`.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`.
|
||||
Rollback: [[frank-large-unsloth-qwen38-27b-q8xl]] (the b11030 prebuilt and
|
||||
the payload are both intact on disk).
|
||||
Rollback, two ways: **payload only** — repoint `MODEL_FILE` at
|
||||
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`,
|
||||
intact on disk. **Engine** — [[frank-large-unsloth-qwen38-27b-q8xl]], the
|
||||
b11030 prebuilt at `/home/dank/GitPackages/unsloth-llama-b11030/`; the only
|
||||
way to get Qwen-Sharp back.
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
|
||||
Reference in new issue
Block a user