dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record dual-f16 KV refit on frank-default-small (2026-08-14)

Browse Source
This commit is contained in:
Frank committed 2026-08-14 12:59:10 -07:00
1 parent f96aa2b692
commit d72092b2c7
4 files changed
+26 -11
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/Dankserver System Specs.md
+2 -2
View File
@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
### Active Services
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 200000 context each (pool 800000). Of the 40 layers' MoE expert weights, **36 sit on CPU** (`--n-cpu-moe 36`) while the remaining **4 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`q8_0`, V=`f16`. Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~21.05 GB VRAM used, leaving an estimated **~3.0 GiB** headroom (chosen for prefill stability, see [[frank-default-small-benchmark]]). Idle RAM ~32 GB of 62 GB.
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 175104 context each (pool 700416, refitted 2026-08-14 for the dual-f16 KV cache). Of the 40 layers' MoE expert weights, **36 sit on CPU** (`--n-cpu-moe 36`) while the remaining **4 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`f16`, V=`f16` (13680 MiB pool at 700416 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~22.1 GB VRAM used, leaving ~2.0 GiB headroom. Idle RAM ~32 GB of 62 GB.
## Hardware Relevance
- 24 GB VRAM is close to saturated with this config. At idle ~21.05 GB used, leaving an estimated **~3.0 GiB** headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072`. A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). By editing `scripts/frank-default-small.env`.
- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). By editing `scripts/frank-default-small.env`.
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).