diff --git a/docs/DFlash DSpark Speculative Decoding.md b/docs/DFlash DSpark Speculative Decoding.md index 8cb31b8..8e948f2 100644 --- a/docs/DFlash DSpark Speculative Decoding.md +++ b/docs/DFlash DSpark Speculative Decoding.md @@ -162,15 +162,17 @@ docker run --rm --name vllm-qwen36-dflash \ ## Deployment on Dankclaw (AMD Strix Halo gfx1151) -Dankclaw runs vLLM-ROCm as its primary runtime. The active service (`frank-default-large`, port 7770) serves the Ling-3.0-flash CIRU INT4 (W4A16) model with native **MTP speculative decoding** (K1, lossless). MTP uses a NextN head preserved in the checkpoint, so no separate draft model is needed. +Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with a **DFlash draft model** (`draft-dflash`) on the ROCm (HIP) backend. The earlier vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime (native MTP) was retired because the model had problems. -Ling-3.0-flash facts: +Current facts: -- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active (hybrid linear attention: KDA + MLA). -- The MTP head is served in-band; vLLM detects it and shares the target embedding and lm_head weights with the draft head. -- The earlier ROCmFPX llama.cpp service (raulvidis ROCmFP4 MTP GGUF) was retired because it was defective. vLLM now handles concurrency and tool calling correctly. +- Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive `Q8_K_P` (dense 27B). +- Draft model: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp `Q8_0` (block size 16). +- Spec command: `--spec-type draft-dflash` with `--spec-draft-n-max 8`. +- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `f16` V. +- Context: 3 x 200700 = 602100; batch 2048 / ubatch 512. -If you want DFlash or DSpark on Dankclaw, use a vLLM speculative-config draft that pairs with the target model. DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use. MTP is the proven, lossless path on gfx1151. +Benchmark result: ROCm beats Vulkan, and DFlash-alone generates faster than a DFlash+ngram layer at 4k. See [[frank-default-large-benchmark]] for full numbers and the memory-bound batch/ubatch limits. ## Notes diff --git a/docs/deployment.md b/docs/deployment.md index 68c6f00..c9ba79f 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -100,7 +100,9 @@ curl -sf http://127.0.0.1:7770/health ### Dankclaw — frank-default-large -The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash CIRU INT4 (W4A16) model. It serves under the alias `frank-default-large` on port 7770 with native MTP speculative decoding. The unit file is `scripts/frank-default-large.service` in the repository. The machine symlinks it to `/etc/systemd/system/frank-default-large.service`. The service launches through `scripts/serve-ling3-vllm.sh`, which sources the CIRU runtime and passes the production settings (host `0.0.0.0`, port 7770, gpu memory utilization 0.95). +The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses a **DFlash draft model** (`draft-dflash`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `f16` V on GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 2048 / ubatch 512. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`. + +> Note: this service replaced the retired vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the earlier llama-serve/vLLM units on Dankclaw. See [[frank-default-large-benchmark]] for the benchmark that drove the backend and spec choices. ### Dankserver — frank-default-small diff --git a/docs/frank-default-large-benchmark.md b/docs/frank-default-large-benchmark.md new file mode 100644 index 0000000..5b87bb1 --- /dev/null +++ b/docs/frank-default-large-benchmark.md @@ -0,0 +1,84 @@ +# frank-default-large — Backend and Spec Benchmark + +This page records the benchmarks that chose the runtime for +`frank-default-large` on Dankclaw. Every number came from the live +llama.cpp build on that machine (upstream, b10133+). + +## Summary + +- **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline. +- **Spec**: DFlash draft model only (`draft-dflash`). +- **Context**: 3 parallel slots x 200700 (pool 602100). +- **KV**: `q8_0` K / `f16` V on GPU. +- **Batch**: 2048 / ubatch 512 (memory-bound, see below). + +## Backend Comparison (llama-bench, raw target model, no draft) + +Both backends ran the same target model at 64k context in `llama-bench` +(`-p 65536 -n 256`). + +| Device | Prefill 64k (t/s) | Generation 64k (t/s) | +| -------- | ----------------- | -------------------- | +| ROCm0 | 220.28 | 6.99 | +| Vulkan0 | 72.34 | 7.00 | + +ROCm prefills 3x faster. Generation is a tie in the raw benchmark. + +## Full-Pipeline Comparison (4k context, real served chain) + +These used `bench-pipeline-64k.sh` driving `llama-server` through the API at +4k context, reading the server's own `print_timing` lines. + +| Spec | Backend | Prefill (t/s) | Generation (t/s) | Draft acceptance | +| --------------------------- | ------- | ------------- | ---------------- | ---------------- | +| `draft-dflash` | ROCm | 148.2 | **20.17** | 0.50 | +| `draft-dflash,ngram-mod` | ROCm | 149.9 | 16.93 | 0.42 | +| `ngram-mod` | ROCm | 154.0 | 8.68 | 0.57 | +| `ngram-mod` | Vulkan | 259.6 | 6.90 | 0.17 | + +### Interpretation + +- DFlash alone gives the fastest generation on ROCm (20.17 t/s). +- Layering ngram-mod lowered ROCm generation (16.93 t/s), so it is not + layered in production. llama.cpp *can* layer DFlash + ngram + (`--spec-type draft-dflash,ngram-mod`), but it was slower here. +- Vulkan only works without the DFlash draft. The DFlash `ctx_other` + setup aborts the Vulkan backend (`ggml-backend.cpp:898: pre-allocated + tensor ... cannot run the operation (NONE)`). Vulkan ngram generation is + also slowest (6.90 t/s) with poor acceptance (0.17). + +**Verdict: ROCm + `draft-dflash`.** + +## Memory Constraint at 602100 Context + +At `3 x 200700` context the KV cache alone is ~50 GB and the model is +29.77 GB, leaving only ~16 GB of the ~96 GB ROCm allocation for compute +graphs. The compute graph scales with **ubatch**, so: + +- batch 12288 / ubatch 4096 -> OOM (80.7 GB prefill buffer) +- batch 6144 / ubatch 2048 -> OOM (40.9 GB prefill buffer) +- batch 2048 / ubatch 2048 -> OOM (ubatch 2048 too large) +- batch 2048 / ubatch 512 -> **fits and loads** + +The requested batch 12288 / ubatch 4096 is not feasible at this context +size on this hardware. + +## Command to Reproduce + +Raw backend bench (ROCm shown; swap `-dev` for the other): + +``` +llama-bench -m
.gguf -p 65536 -n 256 -c 65536 -dev ROCm0 -r 1 +``` + +Full-pipeline bench: + +``` +./scripts/bench-pipeline-64k.sh ROCm0 draft-dflash +``` + +## See Also + +- [[deployment]] — deploy and update process +- [[DFlash DSpark Speculative Decoding]] — DFlash / DSpark / layering details +- [[index]] — knowledge base index diff --git a/docs/index.md b/docs/index.md index 0291ba7..e870383 100644 --- a/docs/index.md +++ b/docs/index.md @@ -17,10 +17,10 @@ experts on CPU, port 7770] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] D[gfx1151 APU 30GiB phys / 96GiB GPU-visible] - E[vLLM-ROCm CIRU INT4 + E[llama.cpp DFlash draft frank-default-large] - F[Ling-3.0-flash CIRU INT4 MTP -port 7770] + F[Qwen3.6-27B uncensored Q8_K_P +DFlash draft, port 7770] D --> E --> F end @@ -51,6 +51,7 @@ port 7770] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU +- [[frank-default-large-benchmark]] — backend/spec benchmark that drove the Dankclaw llama.cpp deployment ## Speculative Decoding & Quantization