Files

T

Frank 677b4af2a4 docs: frank-default-small now uses ngram-mod, DFlash retired

- SWAT from DFlash (see ngram-mod benchmarks below); draft model
  not loaded. Active spec: --spec-type ngram-mod n_min=48/n_max=64/n_match=24.
- Update Dankserver specs, deployment.md, DFlash doc to ngram-mod.
- VRAM footprint: 14 GB (was 20 GB), ~10 GB free headroom.

2026-08-11 02:41:33 -07:00

2.4 KiB

Raw Blame History

Dankserver System Specs

Server Overview

Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.

Hardware Specifications

CPU

Property	Value
Architecture	x86_64
Vendor	AuthenticAMD
Model	AMD Ryzen 7 7700X
Cores	8 (1 socket)
NUMA Nodes	1 (node 0, 63503 MB)

GPU

Property	Value
Model	NVIDIA GeForce RTX 3090
VRAM	24576 MiB (24 GB)
Driver	595.84
CUDA Version	13.2
CUDA Toolkit	12.0 (nvcc)

Memory

Property	Value
System RAM	62 GiB
Swap	127 GiB
/dev/shm	32 GiB

Storage

Filesystem	Size	Used	Avail	Use%
/ (nvme0n1p2)	915G	729G	140G	84%
/mnt/entropy (dm-0)	91T	86T	910G	99%

OS and Software

Property	Value
OS	Ubuntu 24.04.4 LTS (Noble Numbat)
Kernel	7.0.0-28-generic
Python	3.12.3

Active Services

The active frank-default-small service (llama.cpp, b10108+) serves the HauhauCS Qwen3.6-35B-A3B uncensored Q6_K_P model on port 7770 with ngram-mod speculative decoding. It runs 3 parallel slots at 200000 context each (pool 600000). All 40 layers' MoE expert weights sit on CPU (--n-cpu-moe 40) while non-expert tensors and the full-attention KV cache stay on GPU (--gpu-layers all, KV K=q8_0, V=f16). It uses batch 3072 / ubatch 1536 with --ctx-checkpoints and --no-mmap. At load it uses about 14 GB of the 24 GB VRAM and about 32 GB of RAM.

Hardware Relevance

24 GB VRAM limits the GPU-resident footprint. With 3 x 200000 context plus batch 3072/1536 activation buffers, the non-expert weights and KV cache consume about 14 GB, leaving about 10 GB of headroom. MoE experts stay on CPU; --no-mmap preloads expert tensors into RAM, eliminating decode page faults (uses ~32 GB of the 62 GB available).
62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
Root disk has approximately 140 GB free. Store the GGUF models under /home/dank/models (about 32 GB total).

2.4 KiB Raw Blame History