Swap frank-default-large to DFlash2 on Vulkan (PR #27342)

Replace the retired CIRU ROCmFPX runtime with upstream llama.cpp PR #27342 DFlash2 on the Vulkan backend: - Target: unsloth Qwen3.8-27B-UD-Q8_K_XL.gguf (31.4 GB, already on disk) - Drafter: z-lab Qwen3.8-27B-DFlash2-Q4_K_M.gguf (block size 8, top-k 16) - Runtime: /home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server (GGML_VULKAN=ON; HIP lacks TOP_K so the stack runs on Vulkan0/RADV) - spec: --spec-type draft-dflash --spec-draft-n-max 8 (clamps to 7) Validated 2026-08-21 on scratch port 52626: mean acceptance 5.27, health + reasoning completion OK. Archive the retired CIRU .env/.serve script in .AGENTTRASH. Update deployment.md, index.md, AGENTS.md; add docs/dflash2-large-vulkan.md.
2026-08-21 09:40:08 -07:00
parent 701e742f9f
commit fd0d9dd3cb
9 changed files with 470 additions and 136 deletions
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
 | Target | Host | Runtime | Service | Port |
 |--------|------|---------|---------|------|
 | Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
-| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 |
+| Dankclaw | dank@dankclaw | upstream llama.cpp PR #27342 DFlash2 (unsloth Qwen3.8-27B-UD-Q8_K_XL + z-lab DFlash2 Q4_K_M drafter, Vulkan, 262K ctx / 1 slot) | frank-default-large | 52625 |

 ## Workflow

@@ -115,19 +115,19 @@ launches through `scripts/serve-frank-small.sh`, which reads tunables from

 ### Dankclaw — frank-default-large

-The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
-`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v2 GGUF + FFN/GDN
-companions) under the alias `frank-default-large` on port 52625. It runs
-`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
-7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
-(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
-(1 slot; concurrent requests queue, the server never preempts a running
-generation), batch 4096 / ubatch 4096, 16
-context checkpoints, 8 GiB
+The active service on Dankclaw is upstream llama.cpp PR #27342 (DFlash2
+support) built with GGML_VULKAN=ON, serving
+`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf` (unsloth Dynamic v3.0,
+31.4 GB) under the alias `frank-default-large` on port 52625. It runs
+`/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server` with a
+z-lab DFlash2 drafter (`/home/dank/models/Qwen3.8-27B-DFlash2-Q4_K_M.gguf`,
+block size 8) via `--spec-type draft-dflash`, on the Vulkan (RADV) backend
+(`Vulkan0`, gfx1151, 113942 MiB). f16 KV for both K and V, full 262144
+context (1 slot), batch 4096 / ubatch 4096, 16 context checkpoints, 8 GiB
 prompt cache, and reasoning in deepseek format. The unit file is
 `scripts/frank-default-large.service`. It launches through
 `scripts/serve-frank-large.sh`, which reads tunables from
-`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]].
+`scripts/frank-default-large.env`. See [[dflash2-large-vulkan]].

 ## Validation

@@ -0,0 +1,170 @@
+---
+type: reference
+title: DFlash2 on Dankclaw (frank-default-large)
+description: unsloth Qwen3.8-27B-UD-Q8_K_XL served by upstream llama.cpp PR #27342 with a z-lab DFlash2 drafter on the Vulkan backend. Install runbook and the frank-default-large service.
+tags:
+  - llama.cpp
+  - vulkan
+  - dankclaw
+  - gfx1151
+  - dflash
+  - dflash2
+  - speculative-decoding
+timestamp: 2026-08-21
+---
+
+# DFlash2 on Dankclaw (frank-default-large)
+
+`frank-default-large` (port 52625) serves
+`unsloth/Qwen3.8-27B-UD-Q8_K_XL` with a DFlash2 drafter from z-lab. The
+runtime is upstream llama.cpp PR #27342 built with `GGML_VULKAN=ON`.
+This deployment replaced the CIRU HIP runtime on 2026-08-21.
+
+## Why Vulkan
+
+DFlash2 selects its draft candidates with a lightweight path selector
+built on the `TOP_K` and `ARGMAX` graph ops. The HIP backend does not
+implement `TOP_K`. That is why the retired CIRU setup routed the MTP
+draft around it with `LLAMA_MTP_CPU_ARGMAX_FASTPATH`.
+
+The PR reference fork (Nathanw1014/strix-halo-llamacpp) validated DFlash2
+on this exact gfx1151 through Vulkan (RADV). Its `test-backend-ops` passed
+the new graph ops:
+
+| Op | Cases |
+|----|-------|
+| TOP_K | 445 |
+| CONCAT | 193 |
+| PAD | 28 |
+| REPEAT | 18 |
+| FILL | 4 |
+
+The whole stack (target + drafter) runs on `Vulkan0`.
+
+## What Runs
+
+| Artifact | Repo / File | Size |
+|----------|-------------|------|
+| Target | `unsloth/Qwen3.8-27B-GGUF` `Qwen3.8-27B-UD-Q8_K_XL.gguf` | 31.4 GB |
+| Drafter | `z-lab/Qwen3.8-27B-DFlash2-GGUF` `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` | 2.0 GB |
+| Runtime | upstream llama.cpp PR #27342, GGML_VULKAN=ON | - |
+
+The target was already on disk. The drafter was added for this deploy.
+Both live in `/home/dank/models/`.
+
+## Drafter Hyperparameters
+
+Read from the drafter GGUF metadata:
+
+| Parameter | Value |
+|-----------|-------|
+| `dflash.block_count` | 5 |
+| `dflash.block_size` | 8 |
+| `dflash.conv_kernel_size` | 2 |
+| `dflash.conv_group_size` | 16 |
+| `dflash.selector_rank` | 256 |
+| `dflash.selector_top_k` | 16 |
+| `dflash.context_length` | 262144 |
+| `dflash.embedding_length` | 5120 |
+| architecture | `dflash` |
+
+The target and drafter share the Qwen3.8 architecture
+(`qwen35`, dense 27B, vocab 248320), so the drafter inherits the target
+tokenizer and LM head.
+
+## Build Runbook (2026-08-21)
+
+Performed on Dankclaw.
+
+1. In `/home/dank/llama.cpp`, register the `ggml` remote and fetch the PR
+   head:
+   ```
+   git -C /home/dank/llama.cpp remote add ggml https://github.com/ggml-org/llama.cpp.git
+   git -C /home/dank/llama.cpp fetch ggml pull/27342/head:pr-27342
+   ```
+2. Add a detached worktree that does not disturb the running build:
+   ```
+   git -C /home/dank/llama.cpp worktree add --detach /home/dank/llama.cpp-dflash2 pr-27342
+   ```
+3. Configure with Vulkan only (glslang and `glslc` must be installed):
+   ```
+   cmake -B /home/dank/llama.cpp-dflash2/build-dflash2 -G Ninja \
+     -DGGML_VULKAN=ON -DGGML_HIP=OFF -DGGML_CUDA=OFF -DGGML_METAL=OFF \
+     -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release \
+     -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF
+   ```
+4. Build the server:
+   ```
+   cmake --build /home/dank/llama.cpp-dflash2/build-dflash2 --target llama-server -j16
+   ```
+5. Verify the Vulkan device:
+   ```
+   /home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server --list-devices
+   ```
+   Output: `Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO)`.
+
+```mermaid
+graph TD
+    A[Fetch PR #27342 head] --> B[Worktree on DFlash2 head]
+    B --> C[Configure Vulkan build]
+    C --> D[Build llama-server]
+    D --> E[Verify Vulkan0 device]
+    E --> F[Add DFlash2 drafter]
+    F --> G[Swap frank-default-large]
+    G --> H[Health check + completion]
+```
+
+## Service
+
+- Unit: `frank-default-large.service`, port 52625
+- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
+- Binary: `/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server`
+
+### Launch profile
+
+- 262144 context (native max), 1 slot, f16 KV (~16 GiB), 16 context
+  checkpoints, 8 GiB prompt cache
+- DFlash2 draft, block size 8, device Vulkan0, drafter fully offloaded
+- `--spec-type draft-dflash --spec-draft-n-max 8`
+  (clamped to block size 8 - 1 = 7 at runtime)
+- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`)
+- `--no-mmap`, flash attention `-fa on`, `--metrics`
+
+### Required environment
+
+No PromptForge variables are needed. The `TEMP`, `HOST`, and
+`GPU_MAX_HEAP_SIZE` naming rule still applies for portability, even though
+this build uses Vulkan, not HIP.
+
+## Validation
+
+The smoke test on 2026-08-21 used a scratch port (52626) and confirmed:
+
+- Both models load and the server reports healthy.
+- A chat completion returns reasoning and content.
+- The DFlash2 drafter engaged:
+  `draft-dflash: #gen drafts = 11, #acc drafts = 10, #mean acc len = 5.27`.
+- The per-position acceptance curve decays along the block
+  `(0.909, 0.818, 0.727, 0.545, 0.545, 0.364, 0.364)`, as expected.
+
+The published mean acceptance for the Q4_K_M drafter is ~5.39, so 5.27
+is close.
+
+## Operational Notes
+
+- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
+- Model load takes about a minute with a warm OS page cache (31 GB target)
+  and longer on a cold cache. Do not kill the process during warmup.
+- `deploy.sh dankclaw` waits 20 minutes for health.
+- The Vulkan device reports 113942 MiB. rocm-smi reports the 64 GiB
+  unified-memory budget. Trust the runtime for the addressable SVM pool.
+- The MTP head of the target GGUF is ignored (`--spec-type draft-dflash`
+  replaces it). The load log warns about unused MTP tensors; this is normal.
+
+## See Also
+
+- [[DFlash DSpark Speculative Decoding]] — DFlash/DSpark design and history
+- [[ciru-activefpx-promptforge]] — the retired CIRU runtime (archived)
+- [[deployment]] — the deployment and update process
+- [[qwen3.8-strix-optimizations]] — llama.cpp Optimizations on this iGPU
+- [[index]] — knowledge base index
@@ -1,8 +1,8 @@
 # vLLM Scripts — Knowledge Base

 Documentation for the LLM inference fleet on two local machines
-(2026-08-18): upstream llama.cpp on Dankserver (frank-default-small) and
-CIRU ROCmFPX llama.cpp on Dankclaw (frank-default-large).
+(2026-08-21): upstream llama.cpp on Dankserver (frank-default-small) and
+upstream llama.cpp PR #27342 DFlash2 on Dankclaw (frank-default-large).

 ## Deployment Topology

@@ -19,10 +19,11 @@ ngram-mod, 36 MoE experts on CPU, 700K ctx]

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        D[gfx1151 iGPU 64 GiB unified memory]
-        E[CIRU ROCmFPX llama.cpp
+        E[upstream llama.cpp PR #27342 DFlash2, Vulkan
 frank-default-large, port 52625]
-        F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge
-262K ctx / 1 slot, native MTP, PromptForge DualView]
+        F[unsloth Qwen3.8-27B-UD-Q8_K_XL +
+z-lab Qwen3.8-27B-DFlash2-Q4_K_M drafter
+262K ctx / 1 slot, DFlash2 spec]
        D --> E --> F
    end

@@ -41,16 +42,21 @@ frank-default-large, port 52625]

 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[ciru-activefpx-promptforge]] — the CIRU runtime on Dankclaw
-  (frank-default-large)
+- [[dflash2-large-vulkan]] — the DFlash2 runtime on Dankclaw
+  (frank-default-large, Vulkan)
 - [[Dankserver]] — the upstream llama.cpp box (frank-default-small)
- [[Dankclaw]] — the CIRU ROCmFPX box (frank-default-large)
+- [[Dankclaw]] — the DFlash2 Vulkan box (frank-default-large)

-## CIRU Reference (Dankclaw)
+## DFlash2 Reference (Dankclaw)

- [[ciru-activefpx-promptforge]] — model, ROCm 7.15 toolchain, build,
+- [[dflash2-large-vulkan]] — DFlash2 model, Vulkan build, build runbook,
  and the frank-default-large service

+## CIRU Reference (retired)
+
+- [[ciru-activefpx-promptforge]] — the retired CIRU ROCmFPX runtime
+  (archived in .AGENTTRASH, 2026-08-21)
+
 ## vLLM Reference (historical)

 - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism