docs: frank-default-small now 200000 ctx/stream, batch 3072/1536, --no-mmap

Sync deployment.md, Dankserver specs, DFlash doc to the tuned config:
ctx reduced to 200000 per stream (pool 600000) to free KV VRAM for the larger
batch; --no-mmap eager RAM load; ctx-checkpoints 4; ~20GB VRAM / 32GB RAM.
models.json contextWindow -> 200000.
This commit is contained in:
Frank committed 2026-08-11 01:52:12 -07:00
1 parent 9971f83e8b
commit 5c9e812221
3 files changed
+4 -4

No files matched your search

+2 -2
View File
@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
### Active Services
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 262144 context each. All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). At load it uses about 22 GB of the 24 GB VRAM and roughly 4 GB of resident RAM (experts are lazy-mapped).
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 200000 context each (pool 600000). All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). It uses batch 3072 / ubatch 1536 with `--ctx-checkpoints` and `--no-mmap`. At load it uses about 20 GB of the 24 GB VRAM and about 32 GB of RAM.
## Hardware Relevance
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 262144 context, the full-attention KV cache plus non-expert weights consume about 22 GB, leaving about 2 GB of headroom. MoE experts must stay on CPU.
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 200000 context, the full-attention KV cache plus non-expert weights and the batch 3072/1536 activation buffers consume about 20 GB, leaving about 4 GB of headroom. MoE experts must stay on CPU.
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).