docs: frank-default-large llama.cpp draft-dflash deployment + benchmark results; retire Ling vLLM references
This commit is contained in:
1 parent
668083fe76
commit
e92c72fd11
4 files changed
+99
-10
No files matched your search
@@ -162,15 +162,17 @@ docker run --rm --name vllm-qwen36-dflash \
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
Dankclaw runs vLLM-ROCm as its primary runtime. The active service (`frank-default-large`, port 7770) serves the Ling-3.0-flash CIRU INT4 (W4A16) model with native **MTP speculative decoding** (K1, lossless). MTP uses a NextN head preserved in the checkpoint, so no separate draft model is needed.
|
||||
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with a **DFlash draft model** (`draft-dflash`) on the ROCm (HIP) backend. The earlier vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime (native MTP) was retired because the model had problems.
|
||||
|
||||
Ling-3.0-flash facts:
|
||||
Current facts:
|
||||
|
||||
- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active (hybrid linear attention: KDA + MLA).
|
||||
- The MTP head is served in-band; vLLM detects it and shares the target embedding and lm_head weights with the draft head.
|
||||
- The earlier ROCmFPX llama.cpp service (raulvidis ROCmFP4 MTP GGUF) was retired because it was defective. vLLM now handles concurrency and tool calling correctly.
|
||||
- Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive `Q8_K_P` (dense 27B).
|
||||
- Draft model: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp `Q8_0` (block size 16).
|
||||
- Spec command: `--spec-type draft-dflash` with `--spec-draft-n-max 8`.
|
||||
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `f16` V.
|
||||
- Context: 3 x 200700 = 602100; batch 2048 / ubatch 512.
|
||||
|
||||
If you want DFlash or DSpark on Dankclaw, use a vLLM speculative-config draft that pairs with the target model. DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use. MTP is the proven, lossless path on gfx1151.
|
||||
Benchmark result: ROCm beats Vulkan, and DFlash-alone generates faster than a DFlash+ngram layer at 4k. See [[frank-default-large-benchmark]] for full numbers and the memory-bound batch/ubatch limits.
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
Reference in new issue
Block a user