dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-default-small now uses ngram-mod, DFlash retired

Browse Source 
- SWAT from DFlash (see ngram-mod benchmarks below); draft model
  not loaded. Active spec: --spec-type ngram-mod n_min=48/n_max=64/n_match=24.
- Update Dankserver specs, deployment.md, DFlash doc to ngram-mod.
- VRAM footprint: 14 GB (was 20 GB), ~10 GB free headroom.
This commit is contained in:
Frank committed 2026-08-11 02:41:33 -07:00
1 parent fd528ab364
commit 677b4af2a4
3 files changed
+9 -5
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/DFlash DSpark Speculative Decoding.md
+5 -1
View File
@@ -115,7 +115,11 @@ The DFlash draft is not a standalone model. It must pair with its target model.
## Deployment on Dankserver (NVIDIA RTX 3090)
The active Dankserver deployment runs **llama.cpp + DFlash** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 with the `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` Q8_0 draft, 3 slots at 200000 context each, and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`.
## Deployment on Dankserver (NVIDIA RTX 3090)
The active Dankserver deployment runs **llama.cpp** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 using **ngram-mod** speculative decoding (n_min=48 / n_max=64 / n_match=24) with 3 slots at 200000 context each and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`.
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-default-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions: