Files

T

Frank 209223cb4f Offload one more MoE expert to CPU on frank-default-small (34→35 of 40)

- N_CPU_MOE 34→35: 5 expert layers on GPU instead of 6. Frees ~0.4-0.8 GiB
  VRAM for OOM safety after high-load sessions OOM'd at 32 and 33.
- Sync stale doc references (deployment.md, Dankserver System Specs.md)
  that still claimed 32/8; specs now estimate ~1.8 GiB headroom at 35.

2026-08-13 20:18:20 -07:00

2.7 KiB

Raw Blame History

Dankserver System Specs

Server Overview

Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.

Hardware Specifications

CPU

Property	Value
Architecture	x86_64
Vendor	AuthenticAMD
Model	AMD Ryzen 7 7700X
Cores	8 (1 socket)
NUMA Nodes	1 (node 0, 63503 MB)

GPU

Property	Value
Model	NVIDIA GeForce RTX 3090
VRAM	24576 MiB (24 GB)
Driver	595.84
CUDA Version	13.2
CUDA Toolkit	12.0 (nvcc)

Memory

Property	Value
System RAM	62 GiB
Swap	127 GiB
/dev/shm	32 GiB

Storage

Filesystem	Size	Used	Avail	Use%
/ (nvme0n1p2)	915G	729G	140G	84%
/mnt/entropy (dm-0)	91T	86T	910G	99%

OS and Software

Property	Value
OS	Ubuntu 24.04.4 LTS (Noble Numbat)
Kernel	7.0.0-28-generic
Python	3.12.3

Active Services

The active frank-default-small service (llama.cpp, b10108+) serves the HauhauCS Qwen3.6-35B-A3B uncensored Q6_K_P model on port 7770 with ngram-mod speculative decoding. It runs 4 parallel slots at 200000 context each (pool 800000). Of the 40 layers' MoE expert weights, 35 sit on CPU (--n-cpu-moe 35) while the remaining 5 layers' expert weights are resident on GPU along with all non-expert tensors (--gpu-layers all). KV stays on GPU with K=q8_0, V=f16. Batch 8192 / ubatch 2048, --ctx-checkpoints 4, --no-mmap. At idle ~22.2 GB VRAM used, leaving an estimated ~1.8 GiB headroom (up from ~870 MiB at --n-cpu-moe 32). Idle RAM ~32 GB of 62 GB.

Hardware Relevance

24 GB VRAM is close to saturated with this config. At idle ~22.2 GB used, leaving an estimated ~1.8 GiB headroom at --n-cpu-moe 35 (5 layers' experts on GPU). The move from --n-cpu-moe 32 to 35 traded a little GPU-expert speed for OOM safety after high-load sessions OOM'd at 32 and 33. If headroom proves too thin for production bursts, dial back --n-cpu-moe to 37 (only 3 layers on GPU, ~0.9 GiB more headroom) by editing scripts/frank-default-small.env.
62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
Root disk has approximately 140 GB free. Store the GGUF models under /home/dank/models (about 32 GB total).

2.7 KiB Raw Blame History