Files

T

Frank 3c2db64a78 frank-default-small: ncmoe 32 + ubatch 1024 from llama-bench sweep

Swept ncmoe(30..40) x ubatch(256..4096) on the RTX 3090 (Qwen3.6-35B-A3B
Q6_K_P, pp16384/tg512, fit-target 0). Key findings:
- Generation depends only on ncmoe: 40.9 (ncmoe40) .. 51.1 (ncmoe30);
  ubatch/batch do not affect decode at all.
- Prefill + graph VRAM scale with ubatch; logical batch is irrelevant.
- VRAM model (peak + 11.44GiB KV pool + 2.2GiB overhead) reproduces the
  historical OOMs: ncmoe32-ub2048 sums ~24.4GiB > 24.

Chosen state: ncmoe 32 (8 expert layers on GPU), batch 8192, ubatch 1024.
  gen ~48.7 t/s (+7% vs prior ncmoe 35), prefill ~766 t/s, ~1.0GiB headroom.
Alternates: ncmoe34-ub2048 (gen 46.4, pp 1221, 1.5GiB) for prefill-heavy;
  ncmoe32-ub768 (1.3GiB) if headroom matters most.
Docs: new frank-default-small-benchmark.md; synced deployment.md + specs.

2026-08-13 22:56:24 -07:00

2.8 KiB

Raw Blame History

Dankserver System Specs

Server Overview

Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.

Hardware Specifications

CPU

Property	Value
Architecture	x86_64
Vendor	AuthenticAMD
Model	AMD Ryzen 7 7700X
Cores	8 (1 socket)
NUMA Nodes	1 (node 0, 63503 MB)

GPU

Property	Value
Model	NVIDIA GeForce RTX 3090
VRAM	24576 MiB (24 GB)
Driver	595.84
CUDA Version	13.2
CUDA Toolkit	12.0 (nvcc)

Memory

Property	Value
System RAM	62 GiB
Swap	127 GiB
/dev/shm	32 GiB

Storage

Filesystem	Size	Used	Avail	Use%
/ (nvme0n1p2)	915G	729G	140G	84%
/mnt/entropy (dm-0)	91T	86T	910G	99%

OS and Software

Property	Value
OS	Ubuntu 24.04.4 LTS (Noble Numbat)
Kernel	7.0.0-28-generic
Python	3.12.3

Active Services

The active frank-default-small service (llama.cpp, b10108+) serves the HauhauCS Qwen3.6-35B-A3B uncensored Q6_K_P model on port 7770 with ngram-mod speculative decoding. It runs 4 parallel slots at 200000 context each (pool 800000). Of the 40 layers' MoE expert weights, 32 sit on CPU (--n-cpu-moe 32) while the remaining 8 layers' expert weights are resident on GPU along with all non-expert tensors (--gpu-layers all). KV stays on GPU with K=q8_0, V=f16. Batch 8192 / ubatch 1024, --ctx-checkpoints 4, --no-mmap. At idle ~23.0 GB VRAM used, leaving an estimated ~1.0 GiB headroom (chosen by benchmark, see frank-default-small-benchmark). Idle RAM ~32 GB of 62 GB.

Hardware Relevance

24 GB VRAM is close to saturated with this config. At idle ~23.0 GB used, leaving an estimated ~1.0 GiB headroom at --n-cpu-moe 32 (8 layers' experts on GPU) with ubatch 1024. A llama-bench sweep (frank-default-small-benchmark) chose this state: it maximises decode (~48.7 t/s) while staying above the OOM boundary that ncmoe 32 - ubatch 2048 crossed (24.4 GiB). If headroom proves too thin for production bursts, reduce ubatch to 768 (~1.3 GiB headroom) or dial --n-cpu-moe back to 34 at ubatch 1536 (~2.0 GiB headroom) by editing scripts/frank-default-small.env.
62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
Root disk has approximately 140 GB free. Store the GGUF models under /home/dank/models (about 32 GB total).

2.8 KiB Raw Blame History