docs: MTP wins the DFlash replacement; record parallel premature-EOS incident

- benchmark doc: spec A/B table (MTP 15.3-19.5, MTP+ngram 12-13.6,
  MTP n_max2 12.4-14.3, ngram 6.7-8.7, DFlash retired), the DFlash
  shared-draft-context premature-EOS incident, updated batch/ubatch.
- deployment.md: Dankclaw now draft-mtp n_max 3 + MTP-ONLY-Q8_0 sidecar,
  batch 8192/ubatch 8192; note Dankserver bench drift.
- index.md/AGENTS.md: corrected stale topology (Ling-3.0-flash, DFlash).
- DFlash doc: Dankclaw section rewritten for MTP; DFlash retired.
This commit is contained in:
Frank committed 2026-08-13 23:27:06 -07:00
1 parent a27839519e
commit 8d1d9baa82
5 files changed
+42 -19

No files matched your search

+6 -6
View File
@@ -197,17 +197,17 @@ docker run --rm --name vllm-qwen36-dflash \
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with a **DFlash draft model** (`draft-dflash`) on the ROCm (HIP) backend. The earlier vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime (native MTP) was retired because the model had problems.
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with an **MTP sidecar draft** on the ROCm (HIP) backend. DFlash was retired on 2026-08-14.
Current facts:
- Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive `Q8_K_P` (dense 27B).
- Draft model: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp `Q8_0` (block size 16).
- Spec command: `--spec-type draft-dflash` with `--spec-draft-n-max 8`.
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `f16` V.
- Context: 3 x 200700 = 602100; batch 2048 / ubatch 512.
- Draft model: a4lg/Qwen3.6-27B-MTP-ONLY `Q8_0` (MTP head from the base model, loaded as a sidecar).
- Spec command: `--spec-type draft-mtp` with `--spec-draft-n-max 3`.
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V.
- Context: 3 x 200700 = 602100; batch 8192 / ubatch 8192.
Benchmark result: ROCm beats Vulkan, and DFlash-alone generates faster than a DFlash+ngram layer at 4k. See [[frank-default-large-benchmark]] for full numbers and the memory-bound batch/ubatch limits.
Why DFlash was retired: DFlash causes premature EOS on a generating slot when another task lands with high prompt-cache reuse (LCP ~0.99). See [[frank-default-large-benchmark]] for the incident writeup and the MTP-vs-ngram A/B. The DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) remains in `/home/dank/models/` as reference only.
## Notes