docs: frank-default-small now 200000 ctx/stream, batch 3072/1536, --no-mmap
Sync deployment.md, Dankserver specs, DFlash doc to the tuned config: ctx reduced to 200000 per stream (pool 600000) to free KV VRAM for the larger batch; --no-mmap eager RAM load; ctx-checkpoints 4; ~20GB VRAM / 32GB RAM. models.json contextWindow -> 200000.
This commit is contained in:
1 parent
9971f83e8b
commit
5c9e812221
3 files changed
+4
-4
No files matched your search
@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
|
||||
|
||||
### Active Services
|
||||
|
||||
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 262144 context each. All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). At load it uses about 22 GB of the 24 GB VRAM and roughly 4 GB of resident RAM (experts are lazy-mapped).
|
||||
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 200000 context each (pool 600000). All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). It uses batch 3072 / ubatch 1536 with `--ctx-checkpoints` and `--no-mmap`. At load it uses about 20 GB of the 24 GB VRAM and about 32 GB of RAM.
|
||||
|
||||
## Hardware Relevance
|
||||
|
||||
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 262144 context, the full-attention KV cache plus non-expert weights consume about 22 GB, leaving about 2 GB of headroom. MoE experts must stay on CPU.
|
||||
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 200000 context, the full-attention KV cache plus non-expert weights and the batch 3072/1536 activation buffers consume about 20 GB, leaving about 4 GB of headroom. MoE experts must stay on CPU.
|
||||
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
|
||||
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
|
||||
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
|
||||
|
||||
Reference in new issue
Block a user