dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record --spec-default + Vulkan0 + PARALLEL=2 final state; 262k-ctx ROCm-vs-Vulkan bench

Browse Source
This commit is contained in:
Frank committed 2026-08-15 17:13:18 -07:00
1 parent 5296cf11d7
commit b6f2ee2945
6 files changed
+77 -44
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/DFlash DSpark Speculative Decoding.md
+6 -7
View File
@@ -197,18 +197,17 @@ docker run --rm --name vllm-qwen36-dflash \
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with a **DSpark sidecar draft** on the ROCm (HIP) backend.
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with **no dedicated draft** on the **Vulkan0** backend.
Current facts (2026-08-15):
- Main model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (bundled MTP, unused).
- Draft model: `magnitudedev/Qwen3.8-27B-DSpark-GGUF` `Qwen3.8-27B-DSpark-Q8_0.gguf` (DSpark, 1.36B, block_size 7, loaded as a sidecar via `--spec-draft-model`).
- Spec command: `--spec-type draft-dspark` with `--spec-draft-n-max 7`.
- **PARALLEL=1.** DSpark multi-slot crashes the ROCm backend (`ROCm error: an illegal memory access`) under two concurrent 16k prose prompts. See [[qwen3.8-spec-matrix]] for the incident.
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V.
- Context: 1 slot x 262144; batch 8192 / ubatch 256.
- Spec: llama.cpp **`--spec-default`** preset = ngram-mod (n_match 24 / n_min 48 / n_max 64). No draft model.
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes 5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm wins prefill 119 vs 106 t/s. See [[qwen3.8-spec-matrix]] for the bench table.
- **PARALLEL=2** — passes the 2x32k parallel stream test.
- KV `q8_0` K / `q8_0` V; context 2 x 262144; batch 8192 / ubatch 256.
History: before DSpark, the box ran bundled MTP (`draft-mtp`, auto-detected b10411+, retired 2026-08-15). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) and the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
## Notes