dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2427fc448fe961916f18285288de5a50f78621d8
serverAppConfigs/serverAppConfigs-docs/LLM/frank-dragoon-unsloth-qwen38-27b-q8xl.md
T
Frank 55f87fcdce llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot 
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
2026-10-10 17:03:14 -07:00

5.3 KiB
Raw Blame History

type, title, description, tags, timestamp
type title description tags timestamp
reference stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-dragoon) The RETIRED frank-dragoon runbook (2026-09-21 → 2026-09-28; replaced by frank-dragoon-gufo-qwen38-27b, and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing.
llm
llama.cpp
unsloth
dankclaw
frank-dragoon
strix-halo
gfx1151
q8
dflash2
rocm
q8-kv
2026-09-21

frank-dragoon — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)

RETIRED 2026-09-28, replaced by frank-dragoon-gufo-qwen38-27b (the gufo engine, same weights, same port and alias); the prebuilt is intact on disk as the rollback target. ACTIVE from 2026-09-21, when it replaced frank-dragoon-kairic-edge-qwen38-27b (Kairic Edge IU4 W4A4) and frank-dragoon-unsloth-qwen38-flash-next (the Flash-Next MoE it preceded).

Identity

  • Endpoint: frank-dragoon at 192.168.0.143:7770 (Dankclaw).
  • Runtime: stock llama.cpp — the unsloth b11030 prebuilt (/home/dank/GitPackages/unsloth-llama-b11030/llama-server), ROCm gfx1151. A mainline build, not a fork: it supports the qwen3_5 architecture and the DFlash2 drafter (llama_model_qwen35, llama_model_dflash, common_speculative_impl_draft_dflash).
  • Service shape: root systemd unit scripts/frank-dragoon.service (User=dank), launched by scripts/serve-frank-dragoon.sh with tunables from scripts/frank-dragoon.env.

Shape

  • Model: unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL — a 27B dense model, 31.46 GiB GGUF (sha256-verified by scripts/download-frank-dragoon.sh).
  • Context: one stream, full 262,144 tokens (the model's native window), PARALLEL_SLOTS=1.
  • Placement: all layers on the iGPU (--n-gpu-layers 999). The 27B dense weights (31.46 GiB) + the DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) + compute buffers fit the iGPU's GTT aperture (~114 GiB).

Spec method

DFlash2 (a block-diffusion drafter) at depth 4. The drafter is the z-lab Qwen3.8-27B-DFlash2-Q4_K_M (1.14 GiB), loaded via --model-draft with --spec-type draft-dflash and --spec-draft-device ROCm0. It has a sliding window of 2048, so its draft KV is small (~40 MiB). Greedy verification (--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0) is required.

KV

q8_0 (--cache-type-k q8_0 --cache-type-v q8_0), ~8.6 GiB at 262k. The unsloth b11030 build does not support fp8 KV (allowed: f32/f16/bf16/q8_0/ q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the highest-precision 8-bit option.

Template

The Qwen-Sharp v22.4.0 template is pinned via --chat-template-file (dankserver/frank-medium/qwen-sharp-v22.4.0.jinja) with --jinja and --chat-template-kwargs '{"preserve_thinking":true}'. Without --chat-template-file the binary falls back to the built-in peg-native template, which produces a ~73,600-token prompt for a ~250-token request. preserve_thinking keeps prior thinking blocks across turns for multi-turn agents. The template gates reasoning_effort to {xhigh, medium, low} — the harness folds its levels onto that set in models.yml (reasoningEffortMap).

Payload

scripts/download-frank-dragoon.sh fetches the main GGUF and the DFlash2 drafter, each verified against its declared byte count + sha256 (--verify-only re-checks). The serve script refuses to start if the binary, the main GGUF, or the drafter is missing.

Client contract

  • Alias: frank-dragoon (the id at /v1/models).
  • Reasoning: extracted to reasoning_content (--reasoning auto, --reasoning-format deepseek, unlimited budget).
  • Tools: supported (--jinja is pinned).
  • Vision: none — text-only (no mmproj), matching the prior frank-dragoon.
  • Sampling: greedy by default; a client that sends a temperature gets its own, and the harness applies per-model overrides from models.yml.

Failure modes worth knowing

  • Model load is slow (minutes cold). Health returns 503 while loading; do not kill the process. deploy.sh waits up to 20 minutes (LLM_WAIT).
  • --ngl (double-dash) is invalid in b11030 — use --n-gpu-layers (the serve script uses the long form).
  • The 8-bit UD-Q8_K_XL decodes ~17 t/s vs the prior 4-bit Kairic Edge IU4 at ~22 t/s — the ~23 % larger weight size (31.46 vs 26 GiB) explains the gap; it is not a regression.
  • DFlash2 draft acceptance is text-variety-dependent (0.85 on a trivial one-word answer, 0.519 on a varied explanatory prompt) — do not read a single-request acceptance as the steady state.

Rollback

Payload and runtime for frank-dragoon-kairic-edge-qwen38-27b (Kairic Edge IU4 W4A4) remain intact on disk. See frank-dragoon-unsloth-qwen38-flash-next for the Flash-Next predecessor it replaced on 2026-09-21.

Citations

Reference in New Issue View Git Blame Copy Permalink