docs: MTP wins the DFlash replacement; record parallel premature-EOS incident
- benchmark doc: spec A/B table (MTP 15.3-19.5, MTP+ngram 12-13.6, MTP n_max2 12.4-14.3, ngram 6.7-8.7, DFlash retired), the DFlash shared-draft-context premature-EOS incident, updated batch/ubatch. - deployment.md: Dankclaw now draft-mtp n_max 3 + MTP-ONLY-Q8_0 sidecar, batch 8192/ubatch 8192; note Dankserver bench drift. - index.md/AGENTS.md: corrected stale topology (Ling-3.0-flash, DFlash). - DFlash doc: Dankclaw section rewritten for MTP; DFlash retired.
This commit is contained in:
1 parent
a27839519e
commit
8d1d9baa82
5 files changed
+42
-19
No files matched your search
@@ -7,10 +7,30 @@ llama.cpp build on that machine (upstream, b10133+).
|
||||
## Summary
|
||||
|
||||
- **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline.
|
||||
- **Spec**: DFlash draft model only (`draft-dflash`).
|
||||
- **Spec (2026-08-14)**: MTP sidecar drafter (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`). Replaced DFlash, which was retired because it releases a generating slot prematurely under parallel load (see below).
|
||||
- **Context**: 3 parallel slots x 200700 (pool 602100).
|
||||
- **KV**: `q8_0` K / `f16` V on GPU.
|
||||
- **Batch**: 2048 / ubatch 512 (memory-bound, see below).
|
||||
- **KV**: `q8_0` K / `q8_0` V on GPU.
|
||||
- **Batch**: 8192 / ubatch 8192 (2026-08-14, uses the VRAM headroom; peaks ~65.5 GiB of 96 GiB).
|
||||
|
||||
## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)
|
||||
|
||||
| Spec | Generation (t/s) | Parallel-safe |
|
||||
| ---- | ---------------- | -------------- |
|
||||
| DFlash (draft-dflash, n_max 8) | ~20.2 (from earlier 4k bench) | **NO** - premature EOS |
|
||||
| MTP (draft-mtp, n_max 3) | **15.3 - 19.5** | YES |
|
||||
| MTP + ngram-mod (n_max 3) | 12.0 - 13.6 | YES |
|
||||
| MTP (draft-mtp, n_max 2) | 12.4 - 14.3 | YES |
|
||||
| ngram-mod | 6.7 - 8.7 | YES |
|
||||
|
||||
Verdict: `draft-mtp` at `n_max 3` is the winner. It nearly matches DFlash's speed, beats ngram-mod by ~2x, and does not exhibit the DFlash parallel bug. Layering ngram under MTP (12-13.6 t/s) does not pay off, the same result as layering ngram under DFlash.
|
||||
|
||||
## Incident: DFlash Premature EOS Under Parallel Load (2026-08-14)
|
||||
|
||||
Reproduced three times by the user. When a new task lands on a slot with high prompt-cache reuse (LCP similarity ~0.99, `f_keep` ~0.98) while another slot is generating, the generating slot is released ~1 second later with `finish_reason "stop"` and a truncated response. The client records it as a normal completion (no abort). Server log shows a `slot release` with `truncated = 0` and no `cancel task` line.
|
||||
|
||||
Mechanism: DFlash's draft model uses one shared draft context (`ctx_dft`) across all parallel slots. When the new task's first DFlash draft pass runs after a large prompt-reuse operation, it corrupts the generating slot's draft state, and the target spuriously samples the EOG token. Known class of bug upstream (`server : do not default to multiple slots with speculative decoding`, PR #17017; DFlash multi-slot routing fixes in the beellama.cpp fork).
|
||||
|
||||
Fix: switch away from DFlash. ngram-mod (draftless) and MTP (separate draft machinery) both pass the strict parallel test. MTP was chosen for speed.
|
||||
|
||||
## Backend Comparison (llama-bench, raw target model, no draft)
|
||||
|
||||
|
||||
Reference in new issue
Block a user