docs: frank-default-small now uses ngram-mod, DFlash retired
- SWAT from DFlash (see ngram-mod benchmarks below); draft model not loaded. Active spec: --spec-type ngram-mod n_min=48/n_max=64/n_match=24. - Update Dankserver specs, deployment.md, DFlash doc to ngram-mod. - VRAM footprint: 14 GB (was 20 GB), ~10 GB free headroom.
This commit is contained in:
1 parent
fd528ab364
commit
677b4af2a4
3 files changed
+9
-5
No files matched your search
@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
|
||||
|
||||
### Active Services
|
||||
|
||||
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 200000 context each (pool 600000). All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). It uses batch 3072 / ubatch 1536 with `--ctx-checkpoints` and `--no-mmap`. At load it uses about 20 GB of the 24 GB VRAM and about 32 GB of RAM.
|
||||
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs 3 parallel slots at 200000 context each (pool 600000). All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). It uses batch 3072 / ubatch 1536 with `--ctx-checkpoints` and `--no-mmap`. At load it uses about 14 GB of the 24 GB VRAM and about 32 GB of RAM.
|
||||
|
||||
## Hardware Relevance
|
||||
|
||||
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 200000 context, the full-attention KV cache plus non-expert weights and the batch 3072/1536 activation buffers consume about 20 GB, leaving about 4 GB of headroom. MoE experts must stay on CPU.
|
||||
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 200000 context plus batch 3072/1536 activation buffers, the non-expert weights and KV cache consume about 14 GB, leaving about 10 GB of headroom. MoE experts stay on CPU; `--no-mmap` preloads expert tensors into RAM, eliminating decode page faults (uses ~32 GB of the 62 GB available).
|
||||
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
|
||||
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
|
||||
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
|
||||
|
||||
Reference in new issue
Block a user