docs: MTP wins the DFlash replacement; record parallel premature-EOS incident
- benchmark doc: spec A/B table (MTP 15.3-19.5, MTP+ngram 12-13.6, MTP n_max2 12.4-14.3, ngram 6.7-8.7, DFlash retired), the DFlash shared-draft-context premature-EOS incident, updated batch/ubatch. - deployment.md: Dankclaw now draft-mtp n_max 3 + MTP-ONLY-Q8_0 sidecar, batch 8192/ubatch 8192; note Dankserver bench drift. - index.md/AGENTS.md: corrected stale topology (Ling-3.0-flash, DFlash). - DFlash doc: Dankclaw section rewritten for MTP; DFlash retired.
This commit is contained in:
Frank
@@ -2,7 +2,7 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This project manages deployment scripts for two LLM inference servers. Dankclaw runs a vLLM-ROCm service for Ling-3.0-flash (CIRU INT4). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with a DFlash draft and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process.
|
||||
This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.6-27B (HauhauCS uncensored Q8_K_P) with an MTP sidecar drafter (`draft-mtp`, 3 parallel slots, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md) for the spec decision (DFlash retired 2026-08-14 due to a parallel premature-EOS bug).
|
||||
|
||||
## Git Workflow
|
||||
|
||||
@@ -42,8 +42,8 @@ Target map:
|
||||
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 |
|
||||
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + DFlash, MoE offload) | frank-default-small | 7770 |
|
||||
| Dankclaw | dank@dankclaw | llama.cpp (Q8_K_P + MTP sidecar, `draft-mtp`, 3 slots) | frank-default-large | 7770 |
|
||||
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 |
|
||||
|
||||
## Runtime & Troubleshooting
|
||||
|
||||
@@ -89,6 +89,7 @@ graph TD
|
||||
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
|
||||
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
|
||||
- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding.
|
||||
- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw; use `draft-mtp` (n_max 3) or `ngram-mod`, both parallel-safe. See [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md).
|
||||
|
||||
### Dankclaw memory model
|
||||
|
||||
|
||||
@@ -197,17 +197,17 @@ docker run --rm --name vllm-qwen36-dflash \
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with a **DFlash draft model** (`draft-dflash`) on the ROCm (HIP) backend. The earlier vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime (native MTP) was retired because the model had problems.
|
||||
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with an **MTP sidecar draft** on the ROCm (HIP) backend. DFlash was retired on 2026-08-14.
|
||||
|
||||
Current facts:
|
||||
|
||||
- Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive `Q8_K_P` (dense 27B).
|
||||
- Draft model: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp `Q8_0` (block size 16).
|
||||
- Spec command: `--spec-type draft-dflash` with `--spec-draft-n-max 8`.
|
||||
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `f16` V.
|
||||
- Context: 3 x 200700 = 602100; batch 2048 / ubatch 512.
|
||||
- Draft model: a4lg/Qwen3.6-27B-MTP-ONLY `Q8_0` (MTP head from the base model, loaded as a sidecar).
|
||||
- Spec command: `--spec-type draft-mtp` with `--spec-draft-n-max 3`.
|
||||
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V.
|
||||
- Context: 3 x 200700 = 602100; batch 8192 / ubatch 8192.
|
||||
|
||||
Benchmark result: ROCm beats Vulkan, and DFlash-alone generates faster than a DFlash+ngram layer at 4k. See [[frank-default-large-benchmark]] for full numbers and the memory-bound batch/ubatch limits.
|
||||
Why DFlash was retired: DFlash causes premature EOS on a generating slot when another task lands with high prompt-cache reuse (LCP ~0.99). See [[frank-default-large-benchmark]] for the incident writeup and the MTP-vs-ngram A/B. The DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) remains in `/home/dank/models/` as reference only.
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
+4
-2
@@ -137,14 +137,16 @@ curl -sf http://127.0.0.1:7770/health
|
||||
|
||||
### Dankclaw — frank-default-large
|
||||
|
||||
The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses a **DFlash draft model** (`draft-dflash`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `q8_0` V on GPU (pairing required for Flash Attention on this build), full layer offload (`--gpu-layers all`, `--fit off`), and batch 2048 / ubatch 512. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`.
|
||||
The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses an **MTP sidecar drafter** (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `q8_0` V on GPU (pairing required for Flash Attention on this build), full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 8192 (peak ~65.5 GiB VRAM). The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`.
|
||||
|
||||
> Note: this service replaced the retired vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the earlier llama-serve/vLLM units on Dankclaw. See [[frank-default-large-benchmark]] for the benchmark that drove the backend and spec choices.
|
||||
> Note: MTP replaced DFlash (`draft-dflash`) on 2026-08-14. DFlash retired because it releases a generating slot prematurely under parallel load when another task lands with high prompt-cache reuse. See [[frank-default-large-benchmark]]. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the llama-serve/vLLM units on Dankclaw.
|
||||
|
||||
### Dankserver — frank-default-small
|
||||
|
||||
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 200000 context each (pool 800000), and mmproj vision. **35 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 35`); 5 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`q8_0`, V=`f16`). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
|
||||
|
||||
> Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`).
|
||||
|
||||
> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.
|
||||
|
||||
## Validation
|
||||
|
||||
@@ -7,10 +7,30 @@ llama.cpp build on that machine (upstream, b10133+).
|
||||
## Summary
|
||||
|
||||
- **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline.
|
||||
- **Spec**: DFlash draft model only (`draft-dflash`).
|
||||
- **Spec (2026-08-14)**: MTP sidecar drafter (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`). Replaced DFlash, which was retired because it releases a generating slot prematurely under parallel load (see below).
|
||||
- **Context**: 3 parallel slots x 200700 (pool 602100).
|
||||
- **KV**: `q8_0` K / `f16` V on GPU.
|
||||
- **Batch**: 2048 / ubatch 512 (memory-bound, see below).
|
||||
- **KV**: `q8_0` K / `q8_0` V on GPU.
|
||||
- **Batch**: 8192 / ubatch 8192 (2026-08-14, uses the VRAM headroom; peaks ~65.5 GiB of 96 GiB).
|
||||
|
||||
## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)
|
||||
|
||||
| Spec | Generation (t/s) | Parallel-safe |
|
||||
| ---- | ---------------- | -------------- |
|
||||
| DFlash (draft-dflash, n_max 8) | ~20.2 (from earlier 4k bench) | **NO** - premature EOS |
|
||||
| MTP (draft-mtp, n_max 3) | **15.3 - 19.5** | YES |
|
||||
| MTP + ngram-mod (n_max 3) | 12.0 - 13.6 | YES |
|
||||
| MTP (draft-mtp, n_max 2) | 12.4 - 14.3 | YES |
|
||||
| ngram-mod | 6.7 - 8.7 | YES |
|
||||
|
||||
Verdict: `draft-mtp` at `n_max 3` is the winner. It nearly matches DFlash's speed, beats ngram-mod by ~2x, and does not exhibit the DFlash parallel bug. Layering ngram under MTP (12-13.6 t/s) does not pay off, the same result as layering ngram under DFlash.
|
||||
|
||||
## Incident: DFlash Premature EOS Under Parallel Load (2026-08-14)
|
||||
|
||||
Reproduced three times by the user. When a new task lands on a slot with high prompt-cache reuse (LCP similarity ~0.99, `f_keep` ~0.98) while another slot is generating, the generating slot is released ~1 second later with `finish_reason "stop"` and a truncated response. The client records it as a normal completion (no abort). Server log shows a `slot release` with `truncated = 0` and no `cancel task` line.
|
||||
|
||||
Mechanism: DFlash's draft model uses one shared draft context (`ctx_dft`) across all parallel slots. When the new task's first DFlash draft pass runs after a large prompt-reuse operation, it corrupts the generating slot's draft state, and the target spuriously samples the EOG token. Known class of bug upstream (`server : do not default to multiple slots with speculative decoding`, PR #17017; DFlash multi-slot routing fixes in the beellama.cpp fork).
|
||||
|
||||
Fix: switch away from DFlash. ngram-mod (draftless) and MTP (separate draft machinery) both pass the strict parallel test. MTP was chosen for speed.
|
||||
|
||||
## Backend Comparison (llama-bench, raw target model, no draft)
|
||||
|
||||
|
||||
+5
-5
@@ -8,19 +8,19 @@ Documentation for running vLLM inference on two local machines.
|
||||
graph LR
|
||||
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04]
|
||||
A[RTX 3090 24GB + 62GB RAM]
|
||||
B[llama.cpp DFlash
|
||||
B[llama.cpp ngram-mod
|
||||
frank-default-small]
|
||||
C[Qwen3.6-35B-A3B uncensored Q6_K_P
|
||||
experts on CPU, port 7770]
|
||||
MoE experts on CPU, port 7770]
|
||||
A --> B --> C
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
|
||||
E[llama.cpp DFlash draft
|
||||
E[llama.cpp MTP sidecar
|
||||
frank-default-large]
|
||||
F[Qwen3.6-27B uncensored Q8_K_P
|
||||
DFlash draft, port 7770]
|
||||
MTP-ONLY draft, port 7770]
|
||||
D --> E --> F
|
||||
end
|
||||
|
||||
@@ -51,7 +51,7 @@ DFlash draft, port 7770]
|
||||
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark that drove the Dankclaw llama.cpp deployment
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
|
||||
- [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
|
||||
- [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings
|
||||
|
||||
|
||||
Reference in New Issue
Block a user