serverAppConfigs/docs/frank-default-small-benchmark.md

type, title, description, tags

type	title	description	tags
reference	frank-default-small Benchmark — ncmoe and Batch Tuning	llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.	benchmark llamacpp ncmoe ubatch dankserver

frank-default-small Benchmark — ncmoe and Batch/Ubatch Tuning

This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final --n-cpu-moe and --ubatch-size values for frank-default-small.

Run date: 2026-08-13. Build: llama.cpp 555881ebc (10121). CUDA, 7 threads, --cache-type-k q8_0 --cache-type-v f16, --fit-target 0 (KV auto-sized to fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions.

Headline Findings

Generation does not depend on batch or ubatch

For every ncmoe value, generation throughput stayed flat across all ubatch values from 256 to 4096. The delta between the lowest and highest was under one token per second. Generation depends only on how many MoE expert layers stay on the GPU as ncmoe gets smaller.

ncmoe	Expert layers on GPU	Generation (t/s)
40	0	40.9
38	2	42.6
36	4	44.4
35	5	45.5
34	6	46.4
32	8	48.7
30	10	51.1

Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of generation. Moving from ncmoe 35 to ncmoe 32 adds about 3.2 t/s, or 7 percent.

Prefill and graph memory scale with ubatch

Prefill throughput climbs with ubatch because a prompt splits into prompt / ubatch micro-batches. For ncmoe 35 at 16384 prompt tokens:

ubatch	Prefill (t/s)	Peak graph VRAM (MiB)
256	245	6838
512	416	6852
768	556	7074
1024	710	7330
1536	895	7846
2048	1188	8574

The logical --batch-size does not matter. At a fixed ubatch, batch values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM.

Peak VRAM by ncmoe and ubatch

Peak VRAM (llama-bench, single context) rises both when ncmoe drops (more experts on GPU) and when ubatch rises (bigger compute graph):

ncmoe	ub 512	ub 1024	ub 1536	ub 2048
38	4590	5068	5584	6086
36	6098	6576	7092	7594
35	6852	7330	7846	8574
34	7606	8084	8600	9102
32	9114	9592	10108	10610
30	10622	11100	11616	12118

VRAM Budget Model

The live service holds a fixed 800000-token KV pool plus the mmproj vision tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a 24 GiB budget:

service VRAM = llama-bench peak + KV pool (11.44 GiB) + overhead (2.2 GiB)

This model reproduces the historical OOMs. Configurations whose sum exceeds 24 GiB fail. For example, ncmoe 32 - ubatch 2048 sums to about 24.4 GiB and fails, which matches the recorded OOMs at ncmoe 32 and 33.

Candidate Configurations

Configuration	Gen (t/s)	Prefill (t/s)	Headroom (MiB)	Fit
ncmoe 35, ub 2048	45.5	1188	2035	OK
ncmoe 34, ub 2048	46.4	1221	1507	OK
ncmoe 34, ub 1536	46.5	914	2009	OK
ncmoe 32, ub 1024	48.7	766	1017	OK
ncmoe 32, ub 768	48.8	600	1273	OK
ncmoe 30, ub 1536	51.2	1009	under	OOM

Chosen State

The chosen state balances generation speed against prefill speed and OOM headroom:

--n-cpu-moe 32     # 8 expert layers on GPU
--batch-size   8192
--ubatch-size  1024

This gives generation about 48.7 t/s (7 percent above the previous ncmoe 35) while keeping the box above the OOM boundary. If prefill speed matters more than generation, use ncmoe 34 - ubatch 2048. If stability headroom matters most, use ncmoe 34 - ubatch 1536.

Limitations

llama-bench measures raw decode and prefill. It does not run speculative decoding, parallel slots, or the mmproj vision tower. The live service adds those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the real overhead varies a little with load.

See Also

• Dankserver System Specs
• deployment
• DFlash DSpark Speculative Decoding