Files

T

Frank 918fe4eaec Merge vllmScripts under llm/ (history-preserving import)

The vllmScripts repository (LLM inference fleet: serve scripts, systemd
units, env tunables, deployment runbook, research docs) is merged into
this repository. Its full history is grafted under llm/ with a
two-parent merge commit so no commit is lost.

A follow-up commit reorganizes llm/ into the repo layout:
llm/docs -> docs/LLM, llm/scripts -> scripts, llm/old -> old.

2026-08-22 15:03:03 -07:00

6.0 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

DFlash2 on Dankclaw (frank-default-large) — ARCHIVED

Archived 2026-08-21. This DFlash2 Vulkan deployment was retired the same day it shipped because it was slower than the CIRU runtime it replaced. frank-default-large is back on CIRU. This page stays as the runbook and provenance for the DFlash2 attempt.

frank-default-large (port 52625) briefly served unsloth/Qwen3.8-27B-UD-Q8_K_XL with a DFlash2 drafter from z-lab. The runtime was upstream llama.cpp PR #27342 built with GGML_VULKAN=ON.

Why Vulkan

DFlash2 selects its draft candidates with a lightweight path selector built on the TOP_K and ARGMAX graph ops. The HIP backend does not implement TOP_K. That is why the retired CIRU setup routed the MTP draft around it with LLAMA_MTP_CPU_ARGMAX_FASTPATH.

The PR reference fork (Nathanw1014/strix-halo-llamacpp) validated DFlash2 on this exact gfx1151 through Vulkan (RADV). Its test-backend-ops passed the new graph ops:

Op	Cases
TOP_K	445
CONCAT	193
PAD	28
REPEAT	18
FILL	4

The whole stack (target + drafter) runs on Vulkan0.

What Runs

Artifact	Repo / File	Size
Target	`unsloth/Qwen3.8-27B-GGUF` `Qwen3.8-27B-UD-Q8_K_XL.gguf`	31.4 GB
Drafter	`z-lab/Qwen3.8-27B-DFlash2-GGUF` `Qwen3.8-27B-DFlash2-Q4_K_M.gguf`	2.0 GB
Runtime	upstream llama.cpp PR #27342, GGML_VULKAN=ON	-

The target was already on disk. The drafter was added for this deploy. Both live in /home/dank/models/.

Drafter Hyperparameters

Read from the drafter GGUF metadata:

Parameter	Value
`dflash.block_count`	5
`dflash.block_size`	8
`dflash.conv_kernel_size`	2
`dflash.conv_group_size`	16
`dflash.selector_rank`	256
`dflash.selector_top_k`	16
`dflash.context_length`	262144
`dflash.embedding_length`	5120
architecture	`dflash`

The target and drafter share the Qwen3.8 architecture (qwen35, dense 27B, vocab 248320), so the drafter inherits the target tokenizer and LM head.

Build Runbook (2026-08-21)

Performed on Dankclaw.

In /home/dank/llama.cpp, register the ggml remote and fetch the PR head:

git -C /home/dank/llama.cpp remote add ggml https://github.com/ggml-org/llama.cpp.git
git -C /home/dank/llama.cpp fetch ggml pull/27342/head:pr-27342

Add a detached worktree that does not disturb the running build:

git -C /home/dank/llama.cpp worktree add --detach /home/dank/llama.cpp-dflash2 pr-27342

Configure with Vulkan only (glslang and glslc must be installed):

cmake -B /home/dank/llama.cpp-dflash2/build-dflash2 -G Ninja \
  -DGGML_VULKAN=ON -DGGML_HIP=OFF -DGGML_CUDA=OFF -DGGML_METAL=OFF \
  -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF

Build the server:

cmake --build /home/dank/llama.cpp-dflash2/build-dflash2 --target llama-server -j16

Verify the Vulkan device:
```
/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server --list-devices
```
Output: Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO).

graph TD
    A[Fetch PR #27342 head] --> B[Worktree on DFlash2 head]
    B --> C[Configure Vulkan build]
    C --> D[Build llama-server]
    D --> E[Verify Vulkan0 device]
    E --> F[Add DFlash2 drafter]
    F --> G[Swap frank-default-large]
    G --> H[Health check + completion]

Service

Unit: frank-default-large.service, port 52625
Script: scripts/serve-frank-large.sh, env scripts/frank-default-large.env
Binary: /home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server

Launch profile

262144 context (native max), 1 slot, f16 KV (~16 GiB), 16 context checkpoints, 8 GiB prompt cache
DFlash2 draft, block size 8, device Vulkan0, drafter fully offloaded
--spec-type draft-dflash --spec-draft-n-max 8 (clamped to block size 8 - 1 = 7 at runtime)
--temp 0.7 --top-p 0.95, reasoning on (--reasoning-format deepseek)
--no-mmap, flash attention -fa on, --metrics

Required environment

No PromptForge variables are needed. The TEMP, HOST, and GPU_MAX_HEAP_SIZE naming rule still applies for portability, even though this build uses Vulkan, not HIP.

Validation

The smoke test on 2026-08-21 used a scratch port (52626) and confirmed:

Both models load and the server reports healthy.
A chat completion returns reasoning and content.
The DFlash2 drafter engaged: draft-dflash: #gen drafts = 11, #acc drafts = 10, #mean acc len = 5.27.
The per-position acceptance curve decays along the block (0.909, 0.818, 0.727, 0.545, 0.545, 0.364, 0.364), as expected.

The published mean acceptance for the Q4_K_M drafter is ~5.39, so 5.27 is close.

Operational Notes

Health: curl http://127.0.0.1:52625/health. Returns 503 while loading.
Model load takes about a minute with a warm OS page cache (31 GB target) and longer on a cold cache. Do not kill the process during warmup.
deploy.sh dankclaw waits 20 minutes for health.
The Vulkan device reports 113942 MiB. rocm-smi reports the 64 GiB unified-memory budget. Trust the runtime for the addressable SVM pool.
The MTP head of the target GGUF is ignored (--spec-type draft-dflash replaces it). The load log warns about unused MTP tensors; this is normal.

6.0 KiB Raw Blame History