docs: MTP wins the DFlash replacement; record parallel premature-EOS incident
- benchmark doc: spec A/B table (MTP 15.3-19.5, MTP+ngram 12-13.6, MTP n_max2 12.4-14.3, ngram 6.7-8.7, DFlash retired), the DFlash shared-draft-context premature-EOS incident, updated batch/ubatch. - deployment.md: Dankclaw now draft-mtp n_max 3 + MTP-ONLY-Q8_0 sidecar, batch 8192/ubatch 8192; note Dankserver bench drift. - index.md/AGENTS.md: corrected stale topology (Ling-3.0-flash, DFlash). - DFlash doc: Dankclaw section rewritten for MTP; DFlash retired.
This commit is contained in:
1 parent
a27839519e
commit
8d1d9baa82
5 files changed
+42
-19
No files matched your search
@@ -197,17 +197,17 @@ docker run --rm --name vllm-qwen36-dflash \
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with a **DFlash draft model** (`draft-dflash`) on the ROCm (HIP) backend. The earlier vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime (native MTP) was retired because the model had problems.
|
||||
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with an **MTP sidecar draft** on the ROCm (HIP) backend. DFlash was retired on 2026-08-14.
|
||||
|
||||
Current facts:
|
||||
|
||||
- Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive `Q8_K_P` (dense 27B).
|
||||
- Draft model: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp `Q8_0` (block size 16).
|
||||
- Spec command: `--spec-type draft-dflash` with `--spec-draft-n-max 8`.
|
||||
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `f16` V.
|
||||
- Context: 3 x 200700 = 602100; batch 2048 / ubatch 512.
|
||||
- Draft model: a4lg/Qwen3.6-27B-MTP-ONLY `Q8_0` (MTP head from the base model, loaded as a sidecar).
|
||||
- Spec command: `--spec-type draft-mtp` with `--spec-draft-n-max 3`.
|
||||
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V.
|
||||
- Context: 3 x 200700 = 602100; batch 8192 / ubatch 8192.
|
||||
|
||||
Benchmark result: ROCm beats Vulkan, and DFlash-alone generates faster than a DFlash+ngram layer at 4k. See [[frank-default-large-benchmark]] for full numbers and the memory-bound batch/ubatch limits.
|
||||
Why DFlash was retired: DFlash causes premature EOS on a generating slot when another task lands with high prompt-cache reuse (LCP ~0.99). See [[frank-default-large-benchmark]] for the incident writeup and the MTP-vs-ngram A/B. The DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) remains in `/home/dank/models/` as reference only.
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
Reference in new issue
Block a user