frank-large: swap to stock llama.cpp serving Qwen3.8-27B UD-Q8_K_XL with DFlash2

Replace the Kairic Edge ROCmFPX build with the unsloth b11030 stock llama.cpp prebuilt (ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF: UD-Q8_K_XL (a 27B dense model, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU. Serving shape: one 262,144-token conversation, q8_0 KV (the build lacks fp8), text-only, greedy sampling, Qwen-Sharp v22.4.0 template. - scripts/serve-frank-large.sh: stock llama.cpp + DFlash2 spec decode - scripts/frank-large.env: new model/drafter/runtime, 262144 ctx / 1 slot - scripts/download-frank-large.sh: per-file fetch, verify mode, idempotent - scripts/frank-large.service: updated description - scripts/deploy-remote.sh: updated pkill pattern + asset gates - tests/frank_large_unit.bats: new model/runtime/spec assertions - docs: new runbook (active), Kairic Edge runbook retired (rollback target) Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-21 21:50:36 -07:00
1 parent 43b8258061
commit 7689092db8
@@ -0,0 +1,157 @@
+---
+type: reference
+title: stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)
+description: The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, UD-Q8_K_XL) with the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, fp8 KV, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing.
+tags:
+  - llm
+  - llama.cpp
+  - unsloth
+  - dankclaw
+  - frank-large
+  - strix-halo
+  - gfx1151
+  - q8
+  - dflash2
+  - rocm
+  - fp8-kv
+timestamp: 2026-09-21
+---
+
+# stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)
+
+> **ACTIVE since 2026-09-21.** It replaced the Kairic Edge runtime
+> (ciru-ai/ROCmFPX, jcbtc/Qwen3.8-27B-IU4-Kairic-Edge — see
+> [[frank-large-kairic-edge-qwen38-27b]], the current rollback target),
+> which had replaced the unsloth llama.cpp prebuilt serving
+> unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE — see
+> [[frank-large-unsloth-qwen38-flash-next]]). Deployed as a **root
+> systemd unit** (`scripts/frank-large.service`, `User=dank`), port 7770,
+> on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to
+> [[frank-small-gemma4-12b]].
+
+The model is `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a **27B dense**
+Qwen3.8 at **UD-Q8_K_XL** (a high-precision 8-bit quant). It is a qwen3_5
+hybrid: 64 layers total, 16 full attention + 48 Gated DeltaNet linear
+attention. The payload is two files: the 31.46 GiB main GGUF and the 1.14
+GiB DFlash2 Q4_K_M drafter.
+
+The runtime is the **stock llama.cpp** prebuilt (unsloth b11030, ROCm
+gfx1151) at `/home/dank/GitPackages/unsloth-llama-b11030/llama-server`.
+It is a mainline llama.cpp build that supports the qwen3_5 architecture
+and the DFlash2 drafter (llama_model_qwen35, llama_model_dflash,
+common_speculative_impl_draft_dflash).
+
+## Why this swap
+
+- The operator asked for the 27B model at **one conversation, full
+  context (262,144)** with DFlash2 spec decode (2026-09-21).
+- The stock llama.cpp prebuilt is the only stock-compatible runtime with
+  qwen3_5 + DFlash2 support — no source build, no CK `config.h`, no
+  PromptForge sidecars, no compatibility mode.
+- The spec method is DFlash2 (a block-diffusion drafter) at depth 4. The
+  drafter has a sliding window of 2048 (small draft KV, ~40 MiB).
+
+## Shape
+
+One root unit, one process, one published port:
+
+| half | port | reachable |
+|---|---|---|
+| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
+
+## Runtime
+
+- **Prebuilt**: `/home/dank/GitPackages/unsloth-llama-b11030/llama-server`
+  (mainline, ROCm gfx1151 prebuilt, version 0.4.1-dev, build 11030,
+  commit 6ba30d05b). The build bundles its own ROCm runtime — no host SDK
+  gate, no venv.
+- **GPU target**: `gfx1151` (Strix Halo).
+- **Spec-draft flags**: `--spec-draft-type-k`, `--spec-draft-type-v`,
+  `--spec-draft-threads`, `--spec-draft-cpu-mask`, `--spec-draft-prio`,
+  `--spec-draft-poll` all present (confirmed via `--help`).
+
+## Spec method
+
+- **DFlash2 depth 4** (`--spec-type draft-dflash --model-draft <drafter>
+  --spec-draft-n-max 4 --spec-draft-n-min 0`): the block-diffusion
+  drafter proposes 4 tokens per step.
+- **Draft device**: `--spec-draft-device ROCm0` (the drafter runs on the
+  iGPU).
+- **DFlash2 is a block-diffusion drafter** (not a standalone LM). The
+  drafter has a sliding window of 2048 (~40 MiB of draft KV).
+
+## Chat template
+
+The serve script passes `--chat-template-file` pointing at
+`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja` (the Qwen-Sharp
+v22.4.0 template, the same one frank-medium uses). Without this flag the
+binary falls back to the built-in `peg-native` template, which produces a
+~73,600-token prompt for a ~250-token request (280x the expected size).
+With the sharp template wired in, the same request produces a 62–175-token
+prompt. The `Chat format: peg-native` log line is the format-type label,
+not the template indicator — the sharp template is active even when that
+line appears.
+
+## Memory budget
+
+The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
+`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB
+"dedicated" is not the budget). The 27B dense model (31.46 GiB GGUF) is
+mmap'd in with `-ngl 999`; budget at 262k ≈ 41–43 GiB (weights + drafter
+ fp8 KV + buffers), leaving ~71–73 GiB of aperture headroom. The 8 GiB
+prompt-cache budget (`CACHE_RAM=8192`) is the qualified config. The model
+is a 27B hybrid (16 full attention + 48 Gated DeltaNet); its DeltaNet
+state is smaller than a primarily-DeltaNet model, so the fleet's 12.8
+GiB DeltaNet `CACHE_RAM` floor does not apply.
+
+**KV budget**: 16 full-attention layers × 2 (K+V) × 4 kv_heads × 256
+head_dim × 1 byte (fp8) = 32,768 bytes/token. 262,144 tokens × 32,768 ≈
+8 GiB for attention KV. Plus 48 DeltaNet recurrent states (small).
+
+## Payload
+
+`scripts/download-frank-large.sh` (resumable, each file verified against
+its declared byte count + sha256; `--verify-only` re-checks):
+
+| file | size | sha256 |
+|---|---|---|
+| `Qwen3.8-27B-UD-Q8_K_XL.gguf` | 31,457,991,680 | `af36ecb6…` |
+| `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` | 1,143,006,816 | `1a25c568…` |
+
+## Client contract
+
+- **Port**: 7770 (the same port and alias the prior deployments served).
+- **Alias**: `frank-large`.
+- **Context**: 262,144 (the model's native window), 1 slot.
+- **KV**: fp8 quantized (`--cache-type-k fp8 --cache-type-v fp8`).
+- **Sampling**: greedy (`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) —
+  the DFlash2 drafter needs greedy verification.
+- **Reasoning**: enabled (`--reasoning auto --reasoning-format deepseek
+  --reasoning-budget -1`), extracted to `reasoning_content`.
+- **Vision**: text-only (the model has a vision projector — mmproj-BF16
+  is available — but this deployment serves text-only, matching the prior
+  frank-large). The fleet's image path is [[frank-small-gemma4-12b]]
+  either way.
+- **Reasoning effort**: the Qwen3.8 template gates `reasoning_effort` to
+  {xhigh, medium, low} — the harness folds its levels onto that set in
+  `models.yml` (`reasoningEffortMap`).
+
+## Measured performance
+
+(To be filled in after deploy.)
+
+## Failure modes worth knowing
+
+- **The model has `mtp_num_hidden_layers: 1`** but this deployment uses
+  DFlash2 (not MTP) for spec decode. The MTP layer is unused.
+- **The model has vision** (`language_model_only: false`, image_token_id
+  248056, mmproj-BF16 0.93 GB available) but this deployment serves
+  text-only.
+- **DFlash2 needs greedy verification**: the serve script pins
+  `--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`. A non-greedy request
+  will not be verified correctly by the drafter.
+
+## Rollback
+
+[[frank-large-kairic-edge-qwen38-27b]] — the Kairic Edge payload and
+runtime are intact on disk.
@@ -4,11 +4,11 @@ Documentation for the LLM inference fleet on the two hosts (2026-09-21):
 **vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
 process across both Radeon AI PRO R9700 cards (just1moremodel
 Qwen3.8-27B-Uncensored-MXFP4 + the z-lab DFlash2 drafter, bf16 KV, TP=2 over the
-R4D one-shot allreduce), and TWO endpoints on Dankclaw — the Kairic Edge
-runtime (frank-large, ciru-ai/ROCmFPX v1.2, jcbtc Qwen3.8-27B-IU4-Kairic-Edge
-IU4 W4A4 on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
-HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, port
-7771).
+R4D one-shot allreduce), and TWO endpoints on Dankclaw — stock llama.cpp
+(unsloth b11030, ROCm gfx1151; frank-large, unsloth Qwen3.8-27B UD-Q8_K_XL +
+the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, port 7770) and mainline
+llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on
+the RTX 3090, port 7771).

 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -32,10 +32,10 @@ ONE process on both cards, 262144 ctx, up to 8 seqs]
    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        E2[RTX 3090 (PCIe x4 dock) +
 gfx1151 iGPU 114 GiB GTT]
-        E[Kairic Edge (ciru-ai/ROCmFPX v1.2, gfx1151)
+        E[stock llama.cpp (unsloth b11030, gfx1151)
 frank-large, port 7770]
-        F[Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4)
-262144 ctx / 1 slot, MTP depth 4 + n-gram]
+        F[Qwen3.8-27B UD-Q8_K_XL + DFlash2 Q4_K_M drafter
+262144 ctx / 1 slot, DFlash2 depth 4, q8_0 KV]
        G[mainline llama.cpp 37b53fd
 frank-small, port 7771]
        H[Gemma4-12B QAT Q4_K_M
@@ -96,13 +96,21 @@ frank-small, port 7771]
  (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-large-kairic-edge-qwen38-27b]] — ACTIVE frank-large runbook
-  since 2026-09-21 (root systemd unit, port 7770): the Kairic Edge runtime
-  (ciru-ai/ROCmFPX v1.2, built from source) serving
-  jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense) on the gfx1151
-  iGPU — one full 262,144-token conversation, MTP depth 4 + n-gram drafting,
-  the three PromptForge sidecars, compatibility mode, the 114 GiB GTT memory
-  model, client contract, troubleshooting
+- [[frank-large-unsloth-qwen38-27b-q8xl]] — ACTIVE frank-large runbook
+  since 2026-09-21 (root systemd unit, port 7770): stock llama.cpp
+  (unsloth b11030, ROCm gfx1151) serving
+  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2
+  Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
+  conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
+  v22.4.0 template, the 114 GiB GTT memory model, client contract,
+  troubleshooting
+- [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
+  swap, replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]; rollback
+  target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
+  source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
+  on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
+  n-gram drafting, the three PromptForge sidecars, compatibility mode, the
+  114 GiB GTT memory model, client contract, troubleshooting
 - [[frank-large-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
  swap, replaced by [[frank-large-kairic-edge-qwen38-27b]]; rollback target):
  the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving