From 8d1d9baa82c71c198c29cd55e3fbef449b922b30 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 13 Aug 2026 23:27:06 -0700 Subject: [PATCH] docs: MTP wins the DFlash replacement; record parallel premature-EOS incident - benchmark doc: spec A/B table (MTP 15.3-19.5, MTP+ngram 12-13.6, MTP n_max2 12.4-14.3, ngram 6.7-8.7, DFlash retired), the DFlash shared-draft-context premature-EOS incident, updated batch/ubatch. - deployment.md: Dankclaw now draft-mtp n_max 3 + MTP-ONLY-Q8_0 sidecar, batch 8192/ubatch 8192; note Dankserver bench drift. - index.md/AGENTS.md: corrected stale topology (Ling-3.0-flash, DFlash). - DFlash doc: Dankclaw section rewritten for MTP; DFlash retired. --- AGENTS.md | 7 +++--- docs/DFlash DSpark Speculative Decoding.md | 12 +++++----- docs/deployment.md | 6 +++-- docs/frank-default-large-benchmark.md | 26 +++++++++++++++++++--- docs/index.md | 10 ++++----- 5 files changed, 42 insertions(+), 19 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 461a485..bb4a12b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ ## Overview -This project manages deployment scripts for two LLM inference servers. Dankclaw runs a vLLM-ROCm service for Ling-3.0-flash (CIRU INT4). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with a DFlash draft and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process. +This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.6-27B (HauhauCS uncensored Q8_K_P) with an MTP sidecar drafter (`draft-mtp`, 3 parallel slots, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md) for the spec decision (DFlash retired 2026-08-14 due to a parallel premature-EOS bug). ## Git Workflow @@ -42,8 +42,8 @@ Target map: | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 | -| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + DFlash, MoE offload) | frank-default-small | 7770 | +| Dankclaw | dank@dankclaw | llama.cpp (Q8_K_P + MTP sidecar, `draft-mtp`, 3 slots) | frank-default-large | 7770 | +| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 | ## Runtime & Troubleshooting @@ -89,6 +89,7 @@ graph TD - Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first. - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). - Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding. +- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw; use `draft-mtp` (n_max 3) or `ngram-mod`, both parallel-safe. See [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md). ### Dankclaw memory model diff --git a/docs/DFlash DSpark Speculative Decoding.md b/docs/DFlash DSpark Speculative Decoding.md index f75b968..aab53b3 100644 --- a/docs/DFlash DSpark Speculative Decoding.md +++ b/docs/DFlash DSpark Speculative Decoding.md @@ -197,17 +197,17 @@ docker run --rm --name vllm-qwen36-dflash \ ## Deployment on Dankclaw (AMD Strix Halo gfx1151) -Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with a **DFlash draft model** (`draft-dflash`) on the ROCm (HIP) backend. The earlier vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime (native MTP) was retired because the model had problems. +Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with an **MTP sidecar draft** on the ROCm (HIP) backend. DFlash was retired on 2026-08-14. Current facts: - Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive `Q8_K_P` (dense 27B). -- Draft model: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp `Q8_0` (block size 16). -- Spec command: `--spec-type draft-dflash` with `--spec-draft-n-max 8`. -- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `f16` V. -- Context: 3 x 200700 = 602100; batch 2048 / ubatch 512. +- Draft model: a4lg/Qwen3.6-27B-MTP-ONLY `Q8_0` (MTP head from the base model, loaded as a sidecar). +- Spec command: `--spec-type draft-mtp` with `--spec-draft-n-max 3`. +- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V. +- Context: 3 x 200700 = 602100; batch 8192 / ubatch 8192. -Benchmark result: ROCm beats Vulkan, and DFlash-alone generates faster than a DFlash+ngram layer at 4k. See [[frank-default-large-benchmark]] for full numbers and the memory-bound batch/ubatch limits. +Why DFlash was retired: DFlash causes premature EOS on a generating slot when another task lands with high prompt-cache reuse (LCP ~0.99). See [[frank-default-large-benchmark]] for the incident writeup and the MTP-vs-ngram A/B. The DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) remains in `/home/dank/models/` as reference only. ## Notes diff --git a/docs/deployment.md b/docs/deployment.md index 32b0e05..258ca43 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -137,14 +137,16 @@ curl -sf http://127.0.0.1:7770/health ### Dankclaw — frank-default-large -The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses a **DFlash draft model** (`draft-dflash`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `q8_0` V on GPU (pairing required for Flash Attention on this build), full layer offload (`--gpu-layers all`, `--fit off`), and batch 2048 / ubatch 512. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`. +The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses an **MTP sidecar drafter** (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `q8_0` V on GPU (pairing required for Flash Attention on this build), full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 8192 (peak ~65.5 GiB VRAM). The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`. -> Note: this service replaced the retired vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the earlier llama-serve/vLLM units on Dankclaw. See [[frank-default-large-benchmark]] for the benchmark that drove the backend and spec choices. +> Note: MTP replaced DFlash (`draft-dflash`) on 2026-08-14. DFlash retired because it releases a generating slot prematurely under parallel load when another task lands with high prompt-cache reuse. See [[frank-default-large-benchmark]]. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the llama-serve/vLLM units on Dankclaw. ### Dankserver — frank-default-small The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 200000 context each (pool 800000), and mmproj vision. **35 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 35`); 5 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`q8_0`, V=`f16`). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`. +> Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`). + > Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`. ## Validation diff --git a/docs/frank-default-large-benchmark.md b/docs/frank-default-large-benchmark.md index 5b87bb1..0d1e146 100644 --- a/docs/frank-default-large-benchmark.md +++ b/docs/frank-default-large-benchmark.md @@ -7,10 +7,30 @@ llama.cpp build on that machine (upstream, b10133+). ## Summary - **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline. -- **Spec**: DFlash draft model only (`draft-dflash`). +- **Spec (2026-08-14)**: MTP sidecar drafter (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`). Replaced DFlash, which was retired because it releases a generating slot prematurely under parallel load (see below). - **Context**: 3 parallel slots x 200700 (pool 602100). -- **KV**: `q8_0` K / `f16` V on GPU. -- **Batch**: 2048 / ubatch 512 (memory-bound, see below). +- **KV**: `q8_0` K / `q8_0` V on GPU. +- **Batch**: 8192 / ubatch 8192 (2026-08-14, uses the VRAM headroom; peaks ~65.5 GiB of 96 GiB). + +## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt) + +| Spec | Generation (t/s) | Parallel-safe | +| ---- | ---------------- | -------------- | +| DFlash (draft-dflash, n_max 8) | ~20.2 (from earlier 4k bench) | **NO** - premature EOS | +| MTP (draft-mtp, n_max 3) | **15.3 - 19.5** | YES | +| MTP + ngram-mod (n_max 3) | 12.0 - 13.6 | YES | +| MTP (draft-mtp, n_max 2) | 12.4 - 14.3 | YES | +| ngram-mod | 6.7 - 8.7 | YES | + +Verdict: `draft-mtp` at `n_max 3` is the winner. It nearly matches DFlash's speed, beats ngram-mod by ~2x, and does not exhibit the DFlash parallel bug. Layering ngram under MTP (12-13.6 t/s) does not pay off, the same result as layering ngram under DFlash. + +## Incident: DFlash Premature EOS Under Parallel Load (2026-08-14) + +Reproduced three times by the user. When a new task lands on a slot with high prompt-cache reuse (LCP similarity ~0.99, `f_keep` ~0.98) while another slot is generating, the generating slot is released ~1 second later with `finish_reason "stop"` and a truncated response. The client records it as a normal completion (no abort). Server log shows a `slot release` with `truncated = 0` and no `cancel task` line. + +Mechanism: DFlash's draft model uses one shared draft context (`ctx_dft`) across all parallel slots. When the new task's first DFlash draft pass runs after a large prompt-reuse operation, it corrupts the generating slot's draft state, and the target spuriously samples the EOG token. Known class of bug upstream (`server : do not default to multiple slots with speculative decoding`, PR #17017; DFlash multi-slot routing fixes in the beellama.cpp fork). + +Fix: switch away from DFlash. ngram-mod (draftless) and MTP (separate draft machinery) both pass the strict parallel test. MTP was chosen for speed. ## Backend Comparison (llama-bench, raw target model, no draft) diff --git a/docs/index.md b/docs/index.md index d873046..5a0b5b8 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,19 +8,19 @@ Documentation for running vLLM inference on two local machines. graph LR subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04] A[RTX 3090 24GB + 62GB RAM] - B[llama.cpp DFlash + B[llama.cpp ngram-mod frank-default-small] C[Qwen3.6-35B-A3B uncensored Q6_K_P -experts on CPU, port 7770] +MoE experts on CPU, port 7770] A --> B --> C end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] D[gfx1151 APU 30GiB phys / 96GiB GPU-visible] - E[llama.cpp DFlash draft + E[llama.cpp MTP sidecar frank-default-large] F[Qwen3.6-27B uncensored Q8_K_P -DFlash draft, port 7770] +MTP-ONLY draft, port 7770] D --> E --> F end @@ -51,7 +51,7 @@ DFlash draft, port 7770] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU -- [[frank-default-large-benchmark]] — backend/spec benchmark that drove the Dankclaw llama.cpp deployment +- [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug) - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state - [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings