docs: record --spec-default + Vulkan0 + PARALLEL=2 final state; 262k-ctx ROCm-vs-Vulkan bench
This commit is contained in:
1 parent
5296cf11d7
commit
b6f2ee2945
6 files changed
+77
-44
No files matched your search
+14
-15
@@ -139,22 +139,21 @@ curl -sf http://127.0.0.1:7770/health
|
||||
|
||||
The active service on Dankclaw runs llama.cpp serving the unsloth
|
||||
Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port
|
||||
7770. It uses a **DSpark sidecar draft model** (`draft-dspark`,
|
||||
`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max 7`)
|
||||
for speculative decoding on the **ROCm (HIP)** backend, with **1 parallel
|
||||
slot** at 262144 context, KV cache `q8_0` K / `q8_0` V on GPU, full layer
|
||||
offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 256.
|
||||
The unit file is `scripts/frank-default-large.service`. It launches through
|
||||
`scripts/serve-llama-frank-large.sh`, which reads tunables from
|
||||
`scripts/frank-default-large.env`.
|
||||
7770. It uses llama.cpp's **`--spec-default`** preset (ngram-mod, no draft
|
||||
model) for speculative decoding on the **Vulkan0** backend, with **2
|
||||
parallel slots** at 262144 context each, KV cache `q8_0` K / `q8_0` V on
|
||||
GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 /
|
||||
ubatch 256. The unit file is `scripts/frank-default-large.service`. It
|
||||
launches through `scripts/serve-llama-frank-large.sh`, which reads tunables
|
||||
from `scripts/frank-default-large.env`.
|
||||
|
||||
> Note: DSpark replaced **bundled MTP** (`draft-mtp`, in-model, auto-detected
|
||||
> b10411+) on 2026-08-15 at the user's request. **DSpark is single-slot
|
||||
> only**: two concurrent 16k prose prompts crash the ROCm backend with
|
||||
> `ROCm error: an illegal memory access` (status 6/ABRT) during per-sequence
|
||||
> block sampling, so `PARALLEL=1`. DSpark is the semi-autoregressive
|
||||
> extension of DFlash; DFlash was previously retired here for a
|
||||
> shared-draft-context premature-EOS bug. See [[qwen3.8-spec-matrix]].
|
||||
> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
|
||||
> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
|
||||
> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
|
||||
> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
|
||||
> (5.68 vs 3.83 t/s). See [[qwen3.8-spec-matrix]] for both decisions and
|
||||
> the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU
|
||||
> INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw.
|
||||
|
||||
### Dankserver — frank-default-small
|
||||
|
||||
|
||||
Reference in new issue
Block a user