docs: record --spec-default + Vulkan0 + PARALLEL=2 final state; 262k-ctx ROCm-vs-Vulkan bench

This commit is contained in:
Frank committed 2026-08-15 17:13:18 -07:00
1 parent 5296cf11d7
commit b6f2ee2945
6 files changed
+77 -44

No files matched your search

+14 -15
View File
@@ -139,22 +139,21 @@ curl -sf http://127.0.0.1:7770/health
The active service on Dankclaw runs llama.cpp serving the unsloth
Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port
7770. It uses a **DSpark sidecar draft model** (`draft-dspark`,
`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max 7`)
for speculative decoding on the **ROCm (HIP)** backend, with **1 parallel
slot** at 262144 context, KV cache `q8_0` K / `q8_0` V on GPU, full layer
offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 256.
The unit file is `scripts/frank-default-large.service`. It launches through
`scripts/serve-llama-frank-large.sh`, which reads tunables from
`scripts/frank-default-large.env`.
7770. It uses llama.cpp's **`--spec-default`** preset (ngram-mod, no draft
model) for speculative decoding on the **Vulkan0** backend, with **2
parallel slots** at 262144 context each, KV cache `q8_0` K / `q8_0` V on
GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 /
ubatch 256. The unit file is `scripts/frank-default-large.service`. It
launches through `scripts/serve-llama-frank-large.sh`, which reads tunables
from `scripts/frank-default-large.env`.
> Note: DSpark replaced **bundled MTP** (`draft-mtp`, in-model, auto-detected
> b10411+) on 2026-08-15 at the user's request. **DSpark is single-slot
> only**: two concurrent 16k prose prompts crash the ROCm backend with
> `ROCm error: an illegal memory access` (status 6/ABRT) during per-sequence
> block sampling, so `PARALLEL=1`. DSpark is the semi-autoregressive
> extension of DFlash; DFlash was previously retired here for a
> shared-draft-context premature-EOS bug. See [[qwen3.8-spec-matrix]].
> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
> (5.68 vs 3.83 t/s). See [[qwen3.8-spec-matrix]] for both decisions and
> the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU
> INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw.
### Dankserver — frank-default-small