serverAppConfigs/llm/docs/frank-default-small-benchmark.md

type, title, description, tags

type	title	description	tags
reference	frank-default-small Benchmark — ncmoe and Batch Tuning	llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.	benchmark llamacpp ncmoe ubatch dankserver

frank-default-small Benchmark — ncmoe and Batch/Ubatch Tuning

This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final --n-cpu-moe and --ubatch-size values for frank-default-small.

Run date: 2026-08-13. Build: llama.cpp 555881ebc (10121). CUDA, 7 threads, --cache-type-k q8_0 --cache-type-v f16, --fit-target 0 (KV auto-sized to fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions.

Headline Findings

Generation does not depend on batch or ubatch

For every ncmoe value, generation throughput stayed flat across all ubatch values from 256 to 4096. The delta between the lowest and highest was under one token per second. Generation depends only on how many MoE expert layers stay on the GPU as ncmoe gets smaller.

ncmoe	Expert layers on GPU	Generation (t/s)
40	0	40.9
38	2	42.6
36	4	44.4
35	5	45.5
34	6	46.4
32	8	48.7
30	10	51.1

Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of generation. Moving from ncmoe 35 to ncmoe 32 adds about 3.2 t/s, or 7 percent.

Prefill and graph memory scale with ubatch

Prefill throughput climbs with ubatch because a prompt splits into prompt / ubatch micro-batches. For ncmoe 35 at 16384 prompt tokens:

ubatch	Prefill (t/s)	Peak graph VRAM (MiB)
256	245	6838
512	416	6852
768	556	7074
1024	710	7330
1536	895	7846
2048	1188	8574

The logical --batch-size does not matter. At a fixed ubatch, batch values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM.

Peak VRAM by ncmoe and ubatch

Peak VRAM (llama-bench, single context) rises both when ncmoe drops (more experts on GPU) and when ubatch rises (bigger compute graph):

ncmoe	ub 512	ub 1024	ub 1536	ub 2048
38	4590	5068	5584	6086
36	6098	6576	7092	7594
35	6852	7330	7846	8574
34	7606	8084	8600	9102
32	9114	9592	10108	10610
30	10622	11100	11616	12118

VRAM Budget Model

The live service holds a fixed 700416-token KV pool (f16/f16 since 2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a 24 GiB budget:

service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB)

This model reproduces the historical OOMs. Configurations whose sum exceeds 24 GiB fail. For example, ncmoe 32 - ubatch 2048 sums to about 24.4 GiB and fails, which matches the recorded OOMs at ncmoe 32 and 33. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at 800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the service runs 700416 cells (13,680 MiB).

Candidate Configurations

Configuration	Gen (t/s)	Prefill (t/s)	Headroom (MiB)	Fit
ncmoe 35, ub 2048	45.5	1188	2035	OK
ncmoe 34, ub 2048	46.4	1221	1507	OK
ncmoe 34, ub 1536	46.5	914	2009	OK
ncmoe 32, ub 1024	48.7	766	1017	OK
ncmoe 32, ub 768	48.8	600	1273	OK
ncmoe 30, ub 1536	51.2	1009	under	OOM

Chosen State

Prefill speed is the priority, so the state keeps the top prefill tier and moves one expert layer to CPU for stability:

--n-cpu-moe 36     # 4 expert layers on GPU
--batch-size   8192
--ubatch-size  3072

This gives prefill about 1427 t/s at 16384 prompt tokens, generation about 44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool (2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and removes the context-proportional flash-attn F16-conversion workspace that caused the cuBLAS crash class).

ncmoe 36 came from a live crash rather than this sweep. At ncmoe 35 with ubatch 3072, a big prefill burst failed a cuBLAS call under memory pressure (a cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas), which killed the service. Raising ncmoe one step frees about 0.75 GiB of peak VRAM (the grid shows ncmoe 36 - ubatch 3072 peak 8606 MiB versus 9360 MiB at ncmoe 35) for about 2 percent less prefill and 2 percent less decode.

If a crash still occurs under load, set GGML_CUDA_FORCE_MMQ=1 to bypass the cuBLAS path for quantized matmuls, or drop ubatch, or shrink the context pool.

If decode speed matters more than prefill, swap to ncmoe 32 - ubatch 1024 (generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).

Limitations

llama-bench measures raw decode and prefill. It does not run speculative decoding, parallel slots, or the mmproj vision tower. The live service adds those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the real overhead varies a little with load.

See Also

• Dankserver System Specs
• deployment
• DFlash DSpark Speculative Decoding