dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
df00182d0e1663a1ec3e4314bfde8f47f21ef1f9
serverAppConfigs/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-27b-q8xl.md
T
Frank 93d1181fee docs: OKF v0.1 compliance pass + fact-check + 3 research briefs 
- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM
- Restructure log.md to OKF (## YYYY-MM-DD, newest-first)
- Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change
- Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe)
- Write active runbook bodies from ground truth (frank-large, frank-small)
- Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026)
- Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs
- All 336 tests pass; 136 pages, 0 unresolved wikilinks
2026-09-22 02:09:41 -07:00

5.1 KiB
Raw Blame History

type, title, description, tags, timestamp
type title description tags timestamp
reference stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large) The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing.
llm
llama.cpp
unsloth
dankclaw
frank-large
strix-halo
gfx1151
q8
dflash2
rocm
q8-kv
2026-09-21

frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)

ACTIVE since 2026-09-21. Replaces frank-large-kairic-edge-qwen38-27b (Kairic Edge IU4 W4A4, now the rollback target) and frank-large-unsloth-qwen38-flash-next (the Flash-Next MoE it preceded). Same port (7770), same alias (frank-large), same client contract surface.

Identity

  • Endpoint: frank-large at 192.168.0.143:7770 (Dankclaw).
  • Runtime: stock llama.cpp — the unsloth b11030 prebuilt (/home/dank/GitPackages/unsloth-llama-b11030/llama-server), ROCm gfx1151. A mainline build, not a fork: it supports the qwen3_5 architecture and the DFlash2 drafter (llama_model_qwen35, llama_model_dflash, common_speculative_impl_draft_dflash).
  • Service shape: root systemd unit scripts/frank-large.service (User=dank), launched by scripts/serve-frank-large.sh with tunables from scripts/frank-large.env.

Shape

  • Model: unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL — a 27B dense model, 31.46 GiB GGUF (sha256-verified by scripts/download-frank-large.sh).
  • Context: one stream, full 262,144 tokens (the model's native window), PARALLEL_SLOTS=1.
  • Placement: all layers on the iGPU (--n-gpu-layers 999). The 27B dense weights (31.46 GiB) + the DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) + compute buffers fit the iGPU's GTT aperture (~114 GiB).

Spec method

DFlash2 (a block-diffusion drafter) at depth 4. The drafter is the z-lab Qwen3.8-27B-DFlash2-Q4_K_M (1.14 GiB), loaded via --model-draft with --spec-type draft-dflash and --spec-draft-device ROCm0. It has a sliding window of 2048, so its draft KV is small (~40 MiB). Greedy verification (--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0) is required.

KV

q8_0 (--cache-type-k q8_0 --cache-type-v q8_0), ~8.6 GiB at 262k. The unsloth b11030 build does not support fp8 KV (allowed: f32/f16/bf16/q8_0/ q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the highest-precision 8-bit option.

Template

The Qwen-Sharp v22.4.0 template is pinned via --chat-template-file (dankserver/frank-medium/qwen-sharp-v22.4.0.jinja) with --jinja and --chat-template-kwargs '{"preserve_thinking":true}'. Without --chat-template-file the binary falls back to the built-in peg-native template, which produces a ~73,600-token prompt for a ~250-token request. preserve_thinking keeps prior thinking blocks across turns for multi-turn agents. The template gates reasoning_effort to {xhigh, medium, low} — the harness folds its levels onto that set in models.yml (reasoningEffortMap).

Payload

scripts/download-frank-large.sh fetches the main GGUF and the DFlash2 drafter, each verified against its declared byte count + sha256 (--verify-only re-checks). The serve script refuses to start if the binary, the main GGUF, or the drafter is missing.

Client contract

  • Alias: frank-large (the id at /v1/models).
  • Reasoning: extracted to reasoning_content (--reasoning auto, --reasoning-format deepseek, unlimited budget).
  • Tools: supported (--jinja is pinned).
  • Vision: none — text-only (no mmproj), matching the prior frank-large.
  • Sampling: greedy by default; a client that sends a temperature gets its own, and the harness applies per-model overrides from models.yml.

Failure modes worth knowing

  • Model load is slow (minutes cold). Health returns 503 while loading; do not kill the process. deploy.sh waits up to 20 minutes (LLM_WAIT).
  • --ngl (double-dash) is invalid in b11030 — use --n-gpu-layers (the serve script uses the long form).
  • The 8-bit UD-Q8_K_XL decodes ~17 t/s vs the prior 4-bit Kairic Edge IU4 at ~22 t/s — the ~23 % larger weight size (31.46 vs 26 GiB) explains the gap; it is not a regression.
  • DFlash2 draft acceptance is text-variety-dependent (0.85 on a trivial one-word answer, 0.519 on a varied explanatory prompt) — do not read a single-request acceptance as the steady state.

Rollback

Payload and runtime for frank-large-kairic-edge-qwen38-27b (Kairic Edge IU4 W4A4) remain intact on disk. See frank-large-unsloth-qwen38-flash-next for the Flash-Next predecessor it replaced on 2026-09-21.

Citations

Reference in new issue View Git Blame Copy Permalink