Frank
93d1181fee
- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM - Restructure log.md to OKF (## YYYY-MM-DD, newest-first) - Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change - Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe) - Write active runbook bodies from ground truth (frank-large, frank-small) - Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026) - Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs - All 336 tests pass; 136 pages, 0 unresolved wikilinks
116 lines
5.1 KiB
Markdown
116 lines
5.1 KiB
Markdown
---
|
|
type: reference
|
|
title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
|
description: "The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
|
|
tags:
|
|
- llm
|
|
- llama.cpp
|
|
- unsloth
|
|
- dankclaw
|
|
- frank-large
|
|
- strix-halo
|
|
- gfx1151
|
|
- q8
|
|
- dflash2
|
|
- rocm
|
|
- q8-kv
|
|
timestamp: 2026-09-21
|
|
---
|
|
|
|
# frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
|
|
|
|
> **ACTIVE since 2026-09-21.** Replaces [[frank-large-kairic-edge-qwen38-27b]]
|
|
> (Kairic Edge IU4 W4A4, now the rollback target) and
|
|
> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it preceded).
|
|
> Same port (7770), same alias (`frank-large`), same client contract surface.
|
|
|
|
## Identity
|
|
|
|
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
|
- **Runtime:** stock llama.cpp — the unsloth `b11030` prebuilt
|
|
(`/home/dank/GitPackages/unsloth-llama-b11030/llama-server`), ROCm gfx1151.
|
|
A mainline build, not a fork: it supports the `qwen3_5` architecture and the
|
|
DFlash2 drafter (`llama_model_qwen35`, `llama_model_dflash`,
|
|
`common_speculative_impl_draft_dflash`).
|
|
- **Service shape:** root systemd unit `scripts/frank-large.service`
|
|
(`User=dank`), launched by `scripts/serve-frank-large.sh` with tunables from
|
|
`scripts/frank-large.env`.
|
|
|
|
## Shape
|
|
|
|
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a 27B **dense** model,
|
|
31.46 GiB GGUF (sha256-verified by `scripts/download-frank-large.sh`).
|
|
- **Context:** one stream, full 262,144 tokens (the model's native window),
|
|
`PARALLEL_SLOTS=1`.
|
|
- **Placement:** all layers on the iGPU (`--n-gpu-layers 999`). The 27B dense
|
|
weights (31.46 GiB) + the DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) +
|
|
compute buffers fit the iGPU's GTT aperture (~114 GiB).
|
|
|
|
## Spec method
|
|
|
|
DFlash2 (a block-diffusion drafter) at **depth 4**. The drafter is the z-lab
|
|
`Qwen3.8-27B-DFlash2-Q4_K_M` (1.14 GiB), loaded via `--model-draft` with
|
|
`--spec-type draft-dflash` and `--spec-draft-device ROCm0`. It has a sliding
|
|
window of 2048, so its draft KV is small (~40 MiB). Greedy verification
|
|
(`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) is required.
|
|
|
|
## KV
|
|
|
|
**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), ~8.6 GiB at 262k. The
|
|
unsloth b11030 build does **not** support fp8 KV (allowed: f32/f16/bf16/q8_0/
|
|
q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the highest-precision 8-bit option.
|
|
|
|
## Template
|
|
|
|
The Qwen-Sharp v22.4.0 template is pinned via `--chat-template-file`
|
|
(`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja`) with `--jinja` and
|
|
`--chat-template-kwargs '{"preserve_thinking":true}'`. Without
|
|
`--chat-template-file` the binary falls back to the built-in `peg-native`
|
|
template, which produces a ~73,600-token prompt for a ~250-token request.
|
|
`preserve_thinking` keeps prior thinking blocks across turns for multi-turn
|
|
agents. The template gates `reasoning_effort` to `{xhigh, medium, low}` — the
|
|
harness folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
|
|
|
|
## Payload
|
|
|
|
`scripts/download-frank-large.sh` fetches the main GGUF and the DFlash2
|
|
drafter, each verified against its declared byte count + sha256
|
|
(`--verify-only` re-checks). The serve script refuses to start if the binary,
|
|
the main GGUF, or the drafter is missing.
|
|
|
|
## Client contract
|
|
|
|
- **Alias:** `frank-large` (the `id` at `/v1/models`).
|
|
- **Reasoning:** extracted to `reasoning_content` (`--reasoning auto`,
|
|
`--reasoning-format deepseek`, unlimited budget).
|
|
- **Tools:** supported (`--jinja` is pinned).
|
|
- **Vision:** none — text-only (no mmproj), matching the prior frank-large.
|
|
- **Sampling:** greedy by default; a client that sends a temperature gets its
|
|
own, and the harness applies per-model overrides from `models.yml`.
|
|
|
|
## Failure modes worth knowing
|
|
|
|
- **Model load is slow (minutes cold).** Health returns 503 while loading; do
|
|
not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`).
|
|
- **`--ngl` (double-dash) is invalid in b11030** — use `--n-gpu-layers`
|
|
(the serve script uses the long form).
|
|
- **The 8-bit UD-Q8_K_XL decodes ~17 t/s** vs the prior 4-bit Kairic Edge IU4
|
|
at ~22 t/s — the ~23 % larger weight size (31.46 vs 26 GiB) explains the
|
|
gap; it is not a regression.
|
|
- **DFlash2 draft acceptance is text-variety-dependent** (0.85 on a trivial
|
|
one-word answer, 0.519 on a varied explanatory prompt) — do not read a
|
|
single-request acceptance as the steady state.
|
|
|
|
## Rollback
|
|
|
|
Payload and runtime for [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge
|
|
IU4 W4A4) remain intact on disk. See [[frank-large-unsloth-qwen38-flash-next]]
|
|
for the Flash-Next predecessor it replaced on 2026-09-21.
|
|
|
|
# Citations
|
|
|
|
- unsloth prebuilts: https://huggingface.co/unsloth
|
|
- Model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
|
|
- DFlash2 drafter: https://huggingface.co/z-lab
|
|
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
|