docs: MTP wins the DFlash replacement; record parallel premature-EOS incident
- benchmark doc: spec A/B table (MTP 15.3-19.5, MTP+ngram 12-13.6, MTP n_max2 12.4-14.3, ngram 6.7-8.7, DFlash retired), the DFlash shared-draft-context premature-EOS incident, updated batch/ubatch. - deployment.md: Dankclaw now draft-mtp n_max 3 + MTP-ONLY-Q8_0 sidecar, batch 8192/ubatch 8192; note Dankserver bench drift. - index.md/AGENTS.md: corrected stale topology (Ling-3.0-flash, DFlash). - DFlash doc: Dankclaw section rewritten for MTP; DFlash retired.
This commit is contained in:
1 parent
a27839519e
commit
8d1d9baa82
5 files changed
+42
-19
No files matched your search
+4
-2
@@ -137,14 +137,16 @@ curl -sf http://127.0.0.1:7770/health
|
||||
|
||||
### Dankclaw — frank-default-large
|
||||
|
||||
The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses a **DFlash draft model** (`draft-dflash`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `q8_0` V on GPU (pairing required for Flash Attention on this build), full layer offload (`--gpu-layers all`, `--fit off`), and batch 2048 / ubatch 512. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`.
|
||||
The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses an **MTP sidecar drafter** (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `q8_0` V on GPU (pairing required for Flash Attention on this build), full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 8192 (peak ~65.5 GiB VRAM). The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`.
|
||||
|
||||
> Note: this service replaced the retired vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the earlier llama-serve/vLLM units on Dankclaw. See [[frank-default-large-benchmark]] for the benchmark that drove the backend and spec choices.
|
||||
> Note: MTP replaced DFlash (`draft-dflash`) on 2026-08-14. DFlash retired because it releases a generating slot prematurely under parallel load when another task lands with high prompt-cache reuse. See [[frank-default-large-benchmark]]. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the llama-serve/vLLM units on Dankclaw.
|
||||
|
||||
### Dankserver — frank-default-small
|
||||
|
||||
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 200000 context each (pool 800000), and mmproj vision. **35 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 35`); 5 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`q8_0`, V=`f16`). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
|
||||
|
||||
> Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`).
|
||||
|
||||
> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.
|
||||
|
||||
## Validation
|
||||
|
||||
Reference in new issue
Block a user