Files

T

Frank 4c5a4a5f3b docs: frank-default-small — 4 parallel slots, batch 8192/2048, n-cpu-moe 32

Update Dankserver System Specs and deployment to reflect the 4-stream config:
- 4 parallel slots (800000 total ctx), batch 8192 / ubatch 2048
- n-cpu-moe 32 (8 expert layers back on GPU)
- ~870 MiB VRAM headroom under 4-stream load
- Knob documented: n-cpu-moe 36 gives ~1.7 GiB more headroom if needed

2026-08-11 03:13:36 -07:00

2.6 KiB

Raw Blame History

Dankserver System Specs

Server Overview

Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.

Hardware Specifications

CPU

Property	Value
Architecture	x86_64
Vendor	AuthenticAMD
Model	AMD Ryzen 7 7700X
Cores	8 (1 socket)
NUMA Nodes	1 (node 0, 63503 MB)

GPU

Property	Value
Model	NVIDIA GeForce RTX 3090
VRAM	24576 MiB (24 GB)
Driver	595.84
CUDA Version	13.2
CUDA Toolkit	12.0 (nvcc)

Memory

Property	Value
System RAM	62 GiB
Swap	127 GiB
/dev/shm	32 GiB

Storage

Filesystem	Size	Used	Avail	Use%
/ (nvme0n1p2)	915G	729G	140G	84%
/mnt/entropy (dm-0)	91T	86T	910G	99%

OS and Software

Property	Value
OS	Ubuntu 24.04.4 LTS (Noble Numbat)
Kernel	7.0.0-28-generic
Python	3.12.3

Active Services

The active frank-default-small service (llama.cpp, b10108+) serves the HauhauCS Qwen3.6-35B-A3B uncensored Q6_K_P model on port 7770 with ngram-mod speculative decoding. It runs 4 parallel slots at 200000 context each (pool 800000). Of the 40 layers' MoE expert weights, 32 sit on CPU (--n-cpu-moe 32) while the remaining 8 layers' expert weights are resident on GPU along with all non-expert tensors (--gpu-layers all). KV stays on GPU with K=q8_0, V=f16. Batch 8192 / ubatch 2048, --ctx-checkpoints 4, --no-mmap. At idle ~23.5 GB VRAM used; under a 4-stream 300-token burst peaks ~23.7 GB, leaving ~870 MiB headroom. Idle RAM ~32 GB of 62 GB.

Hardware Relevance

24 GB VRAM is almost fully saturated with this config. At idle ~23.5 GB used; under a 4-stream 300-token burst peaks at ~23.7 GB, leaving ~870 MiB headroom. The 8 layers' expert weights back on GPU account for the tight margin. If headroom proves too thin for production bursts, dial back --n-cpu-moe to 36 (only 4 layers on GPU, ~1.7 GiB more headroom) by editing scripts/frank-default-small.env.
62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
Root disk has approximately 140 GB free. Store the GGUF models under /home/dank/models (about 32 GB total).

2.6 KiB Raw Blame History