dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: OKF v0.1 compliance pass + fact-check + 3 research briefs

Browse Source 
- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM
- Restructure log.md to OKF (## YYYY-MM-DD, newest-first)
- Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change
- Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe)
- Write active runbook bodies from ground truth (frank-large, frank-small)
- Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026)
- Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs
- All 336 tests pass; 136 pages, 0 unresolved wikilinks
This commit is contained in:
Frank committed 2026-09-22 02:09:41 -07:00
1 parent 24f7a7772d
commit 93d1181fee
103 files changed
+2047 -4329
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md
+74 -116
View File
@@ -1,7 +1,7 @@
---
type: reference
title: stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)
description: The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, UD-Q8_K_XL) with the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, fp8 KV, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing.
title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
description: "The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
tags:
- llm
- llama.cpp
@@ -13,145 +13,103 @@ tags:
- q8
- dflash2
- rocm
- fp8-kv
- q8-kv
timestamp: 2026-09-21
---
# stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)
# frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
> **ACTIVE since 2026-09-21.** It replaced the Kairic Edge runtime
> (ciru-ai/ROCmFPX, jcbtc/Qwen3.8-27B-IU4-Kairic-Edge — see
> [[frank-large-kairic-edge-qwen38-27b]], the current rollback target),
> which had replaced the unsloth llama.cpp prebuilt serving
> unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE — see
> [[frank-large-unsloth-qwen38-flash-next]]). Deployed as a **root
> systemd unit** (`scripts/frank-large.service`, `User=dank`), port 7770,
> on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to
> [[frank-small-gemma4-12b]].
> **ACTIVE since 2026-09-21.** Replaces [[frank-large-kairic-edge-qwen38-27b]]
> (Kairic Edge IU4 W4A4, now the rollback target) and
> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it preceded).
> Same port (7770), same alias (`frank-large`), same client contract surface.
The model is `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a **27B dense**
Qwen3.8 at **UD-Q8_K_XL** (a high-precision 8-bit quant). It is a qwen3_5
hybrid: 64 layers total, 16 full attention + 48 Gated DeltaNet linear
attention. The payload is two files: the 31.46 GiB main GGUF and the 1.14
GiB DFlash2 Q4_K_M drafter.
## Identity
The runtime is the **stock llama.cpp** prebuilt (unsloth b11030, ROCm
gfx1151) at `/home/dank/GitPackages/unsloth-llama-b11030/llama-server`.
It is a mainline llama.cpp build that supports the qwen3_5 architecture
and the DFlash2 drafter (llama_model_qwen35, llama_model_dflash,
common_speculative_impl_draft_dflash).
## Why this swap
- The operator asked for the 27B model at **one conversation, full
context (262,144)** with DFlash2 spec decode (2026-09-21).
- The stock llama.cpp prebuilt is the only stock-compatible runtime with
qwen3_5 + DFlash2 support — no source build, no CK `config.h`, no
PromptForge sidecars, no compatibility mode.
- The spec method is DFlash2 (a block-diffusion drafter) at depth 4. The
drafter has a sliding window of 2048 (small draft KV, ~40 MiB).
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
- **Runtime:** stock llama.cpp — the unsloth `b11030` prebuilt
(`/home/dank/GitPackages/unsloth-llama-b11030/llama-server`), ROCm gfx1151.
A mainline build, not a fork: it supports the `qwen3_5` architecture and the
DFlash2 drafter (`llama_model_qwen35`, `llama_model_dflash`,
`common_speculative_impl_draft_dflash`).
- **Service shape:** root systemd unit `scripts/frank-large.service`
(`User=dank`), launched by `scripts/serve-frank-large.sh` with tunables from
`scripts/frank-large.env`.
## Shape
One root unit, one process, one published port:
| half | port | reachable |
|---|---|---|
| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
## Runtime
- **Prebuilt**: `/home/dank/GitPackages/unsloth-llama-b11030/llama-server`
(mainline, ROCm gfx1151 prebuilt, version 0.4.1-dev, build 11030,
commit 6ba30d05b). The build bundles its own ROCm runtime — no host SDK
gate, no venv.
- **GPU target**: `gfx1151` (Strix Halo).
- **Spec-draft flags**: `--spec-draft-type-k`, `--spec-draft-type-v`,
`--spec-draft-threads`, `--spec-draft-cpu-mask`, `--spec-draft-prio`,
`--spec-draft-poll` all present (confirmed via `--help`).
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a 27B **dense** model,
31.46 GiB GGUF (sha256-verified by `scripts/download-frank-large.sh`).
- **Context:** one stream, full 262,144 tokens (the model's native window),
`PARALLEL_SLOTS=1`.
- **Placement:** all layers on the iGPU (`--n-gpu-layers 999`). The 27B dense
weights (31.46 GiB) + the DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) +
compute buffers fit the iGPU's GTT aperture (~114 GiB).
## Spec method
- **DFlash2 depth 4** (`--spec-type draft-dflash --model-draft <drafter>
--spec-draft-n-max 4 --spec-draft-n-min 0`): the block-diffusion
drafter proposes 4 tokens per step.
- **Draft device**: `--spec-draft-device ROCm0` (the drafter runs on the
iGPU).
- **DFlash2 is a block-diffusion drafter** (not a standalone LM). The
drafter has a sliding window of 2048 (~40 MiB of draft KV).
DFlash2 (a block-diffusion drafter) at **depth 4**. The drafter is the z-lab
`Qwen3.8-27B-DFlash2-Q4_K_M` (1.14 GiB), loaded via `--model-draft` with
`--spec-type draft-dflash` and `--spec-draft-device ROCm0`. It has a sliding
window of 2048, so its draft KV is small (~40 MiB). Greedy verification
(`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) is required.
## Chat template
## KV
The serve script passes `--chat-template-file` pointing at
`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja` (the Qwen-Sharp
v22.4.0 template, the same one frank-medium uses). Without this flag the
binary falls back to the built-in `peg-native` template, which produces a
~73,600-token prompt for a ~250-token request (280x the expected size).
With the sharp template wired in, the same request produces a 62–175-token
prompt. The `Chat format: peg-native` log line is the format-type label,
not the template indicator — the sharp template is active even when that
line appears.
**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), ~8.6 GiB at 262k. The
unsloth b11030 build does **not** support fp8 KV (allowed: f32/f16/bf16/q8_0/
q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the highest-precision 8-bit option.
## Memory budget
## Template
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB
"dedicated" is not the budget). The 27B dense model (31.46 GiB GGUF) is
mmap'd in with `-ngl 999`; budget at 262k ≈ 41–43 GiB (weights + drafter
+ fp8 KV + buffers), leaving ~71–73 GiB of aperture headroom. The 8 GiB
prompt-cache budget (`CACHE_RAM=8192`) is the qualified config. The model
is a 27B hybrid (16 full attention + 48 Gated DeltaNet); its DeltaNet
state is smaller than a primarily-DeltaNet model, so the fleet's 12.8
GiB DeltaNet `CACHE_RAM` floor does not apply.
**KV budget**: 16 full-attention layers × 2 (K+V) × 4 kv_heads × 256
head_dim × 1 byte (fp8) = 32,768 bytes/token. 262,144 tokens × 32,768 ≈
8 GiB for attention KV. Plus 48 DeltaNet recurrent states (small).
The Qwen-Sharp v22.4.0 template is pinned via `--chat-template-file`
(`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja`) with `--jinja` and
`--chat-template-kwargs '{"preserve_thinking":true}'`. Without
`--chat-template-file` the binary falls back to the built-in `peg-native`
template, which produces a ~73,600-token prompt for a ~250-token request.
`preserve_thinking` keeps prior thinking blocks across turns for multi-turn
agents. The template gates `reasoning_effort` to `{xhigh, medium, low}` — the
harness folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
## Payload
`scripts/download-frank-large.sh` (resumable, each file verified against
its declared byte count + sha256; `--verify-only` re-checks):
| file | size | sha256 |
|---|---|---|
| `Qwen3.8-27B-UD-Q8_K_XL.gguf` | 31,457,991,680 | `af36ecb6…` |
| `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` | 1,143,006,816 | `1a25c568…` |
`scripts/download-frank-large.sh` fetches the main GGUF and the DFlash2
drafter, each verified against its declared byte count + sha256
(`--verify-only` re-checks). The serve script refuses to start if the binary,
the main GGUF, or the drafter is missing.
## Client contract
- **Port**: 7770 (the same port and alias the prior deployments served).
- **Alias**: `frank-large`.
- **Context**: 262,144 (the model's native window), 1 slot.
- **KV**: fp8 quantized (`--cache-type-k fp8 --cache-type-v fp8`).
- **Sampling**: greedy (`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) —
the DFlash2 drafter needs greedy verification.
- **Reasoning**: enabled (`--reasoning auto --reasoning-format deepseek
--reasoning-budget -1`), extracted to `reasoning_content`.
- **Vision**: text-only (the model has a vision projector — mmproj-BF16
is available — but this deployment serves text-only, matching the prior
frank-large). The fleet's image path is [[frank-small-gemma4-12b]]
either way.
- **Reasoning effort**: the Qwen3.8 template gates `reasoning_effort` to
{xhigh, medium, low} — the harness folds its levels onto that set in
`models.yml` (`reasoningEffortMap`).
## Measured performance
(To be filled in after deploy.)
- **Alias:** `frank-large` (the `id` at `/v1/models`).
- **Reasoning:** extracted to `reasoning_content` (`--reasoning auto`,
`--reasoning-format deepseek`, unlimited budget).
- **Tools:** supported (`--jinja` is pinned).
- **Vision:** none — text-only (no mmproj), matching the prior frank-large.
- **Sampling:** greedy by default; a client that sends a temperature gets its
own, and the harness applies per-model overrides from `models.yml`.
## Failure modes worth knowing
- **The model has `mtp_num_hidden_layers: 1`** but this deployment uses
DFlash2 (not MTP) for spec decode. The MTP layer is unused.
- **The model has vision** (`language_model_only: false`, image_token_id
248056, mmproj-BF16 0.93 GB available) but this deployment serves
text-only.
- **DFlash2 needs greedy verification**: the serve script pins
`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`. A non-greedy request
will not be verified correctly by the drafter.
- **Model load is slow (minutes cold).** Health returns 503 while loading; do
not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`).
- **`--ngl` (double-dash) is invalid in b11030** — use `--n-gpu-layers`
(the serve script uses the long form).
- **The 8-bit UD-Q8_K_XL decodes ~17 t/s** vs the prior 4-bit Kairic Edge IU4
at ~22 t/s — the ~23 % larger weight size (31.46 vs 26 GiB) explains the
gap; it is not a regression.
- **DFlash2 draft acceptance is text-variety-dependent** (0.85 on a trivial
one-word answer, 0.519 on a varied explanatory prompt) — do not read a
single-request acceptance as the steady state.
## Rollback
[[frank-large-kairic-edge-qwen38-27b]] — the Kairic Edge payload and
runtime are intact on disk.
Payload and runtime for [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge
IU4 W4A4) remain intact on disk. See [[frank-large-unsloth-qwen38-flash-next]]
for the Flash-Next predecessor it replaced on 2026-09-21.
# Citations
- unsloth prebuilts: https://huggingface.co/unsloth
- Model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
- DFlash2 drafter: https://huggingface.co/z-lab
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`