Files

T

Frank 01eb957f90 frank-default-small: prefill-max ncmoe 35 / ubatch 3072

Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
  ~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
  decode cost. Docs synced (benchmark Chosen State, deployment, specs).

2026-08-13 23:07:30 -07:00

2.7 KiB

Raw Blame History

Dankserver System Specs

Server Overview

Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.

Hardware Specifications

CPU

Property	Value
Architecture	x86_64
Vendor	AuthenticAMD
Model	AMD Ryzen 7 7700X
Cores	8 (1 socket)
NUMA Nodes	1 (node 0, 63503 MB)

GPU

Property	Value
Model	NVIDIA GeForce RTX 3090
VRAM	24576 MiB (24 GB)
Driver	595.84
CUDA Version	13.2
CUDA Toolkit	12.0 (nvcc)

Memory

Property	Value
System RAM	62 GiB
Swap	127 GiB
/dev/shm	32 GiB

Storage

Filesystem	Size	Used	Avail	Use%
/ (nvme0n1p2)	915G	729G	140G	84%
/mnt/entropy (dm-0)	91T	86T	910G	99%

OS and Software

Property	Value
OS	Ubuntu 24.04.4 LTS (Noble Numbat)
Kernel	7.0.0-28-generic
Python	3.12.3

Active Services

The active frank-default-small service (llama.cpp, b10108+) serves the HauhauCS Qwen3.6-35B-A3B uncensored Q6_K_P model on port 7770 with ngram-mod speculative decoding. It runs 4 parallel slots at 200000 context each (pool 800000). Of the 40 layers' MoE expert weights, 35 sit on CPU (--n-cpu-moe 35) while the remaining 5 layers' expert weights are resident on GPU along with all non-expert tensors (--gpu-layers all). KV stays on GPU with K=q8_0, V=f16. Batch 8192 / ubatch 3072, --ctx-checkpoints 4, --no-mmap. At idle ~22.8 GB VRAM used, leaving an estimated ~1.2 GiB headroom (chosen by benchmark for prefill speed, see frank-default-small-benchmark). Idle RAM ~32 GB of 62 GB.

Hardware Relevance

24 GB VRAM is close to saturated with this config. At idle ~22.8 GB used, leaving an estimated ~1.2 GiB headroom at --n-cpu-moe 35 (5 layers' experts on GPU) with ubatch 3072. A llama-bench sweep (frank-default-small-benchmark) chose this state because prefill is the priority: ubatch 3072 reaches ~1458 t/s prefill, the top tier, and generation is independent of ubatch. If decode speed matters more, use ncmoe 32 - ubatch 1024 (gen ~48.7 t/s). By editing scripts/frank-default-small.env.
62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
Root disk has approximately 140 GB free. Store the GGUF models under /home/dank/models (about 32 GB total).

2.7 KiB Raw Blame History