dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-default-small now 200000 ctx/stream, batch 3072/1536, --no-mmap

Browse Source 
Sync deployment.md, Dankserver specs, DFlash doc to the tuned config:
ctx reduced to 200000 per stream (pool 600000) to free KV VRAM for the larger
batch; --no-mmap eager RAM load; ctx-checkpoints 4; ~20GB VRAM / 32GB RAM.
models.json contextWindow -> 200000.
This commit is contained in:
Frank committed 2026-08-11 01:52:12 -07:00
1 parent 9971f83e8b
commit 5c9e812221
3 files changed
+4 -4
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/DFlash DSpark Speculative Decoding.md
+1 -1
View File
@@ -115,7 +115,7 @@ The DFlash draft is not a standalone model. It must pair with its target model.
## Deployment on Dankserver (NVIDIA RTX 3090)
The active Dankserver deployment runs **llama.cpp + DFlash** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 with the `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` Q8_0 draft, 3 slots at 262144 context, and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`.
The active Dankserver deployment runs **llama.cpp + DFlash** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 with the `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` Q8_0 draft, 3 slots at 200000 context each, and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`.
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions: