--- type: reference title: frank-default-small Benchmark — ncmoe and Batch Tuning description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P. tags: - benchmark - llamacpp - ncmoe - ubatch - dankserver --- # frank-default-small Benchmark — ncmoe and Batch/Ubatch Tuning This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final `--n-cpu-moe` and `--ubatch-size` values for `frank-default-small`. Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads, `--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions. ## Headline Findings ### Generation does not depend on batch or ubatch For every `ncmoe` value, generation throughput stayed flat across all `ubatch` values from 256 to 4096. The delta between the lowest and highest was under one token per second. Generation depends only on how many MoE expert layers stay on the GPU as `ncmoe` gets smaller. | `ncmoe` | Expert layers on GPU | Generation (t/s) | | ------- | -------------------- | ---------------- | | 40 | 0 | 40.9 | | 38 | 2 | 42.6 | | 36 | 4 | 44.4 | | 35 | 5 | 45.5 | | 34 | 6 | 46.4 | | 32 | 8 | 48.7 | | 30 | 10 | 51.1 | Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of generation. Moving from `ncmoe 35` to `ncmoe 32` adds about 3.2 t/s, or 7 percent. ### Prefill and graph memory scale with ubatch Prefill throughput climbs with `ubatch` because a prompt splits into `prompt / ubatch` micro-batches. For `ncmoe 35` at 16384 prompt tokens: | `ubatch` | Prefill (t/s) | Peak graph VRAM (MiB) | | -------- | ------------- | --------------------- | | 256 | 245 | 6838 | | 512 | 416 | 6852 | | 768 | 556 | 7074 | | 1024 | 710 | 7330 | | 1536 | 895 | 7846 | | 2048 | 1188 | 8574 | The logical `--batch-size` does not matter. At a fixed `ubatch`, `batch` values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM. ## Peak VRAM by ncmoe and ubatch Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops (more experts on GPU) and when `ubatch` rises (bigger compute graph): | `ncmoe` | ub 512 | ub 1024 | ub 1536 | ub 2048 | | ------- | ------ | ------- | ------- | ------- | | 38 | 4590 | 5068 | 5584 | 6086 | | 36 | 6098 | 6576 | 7092 | 7594 | | 35 | 6852 | 7330 | 7846 | 8574 | | 34 | 7606 | 8084 | 8600 | 9102 | | 32 | 9114 | 9592 | 10108 | 10610 | | 30 | 10622 | 11100 | 11616 | 12118 | ## VRAM Budget Model The live service holds a fixed 700416-token KV pool (f16/f16 since 2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a 24 GiB budget: ``` service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB) ``` This model reproduces the historical OOMs. Configurations whose sum exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about 24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and `33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at 800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the service runs 700416 cells (13,680 MiB). ## Candidate Configurations | Configuration | Gen (t/s) | Prefill (t/s) | Headroom (MiB) | Fit | | ------------------ | --------- | ------------- | -------------- | --- | | ncmoe 35, ub 2048 | 45.5 | 1188 | 2035 | OK | | ncmoe 34, ub 2048 | 46.4 | 1221 | 1507 | OK | | ncmoe 34, ub 1536 | 46.5 | 914 | 2009 | OK | | ncmoe 32, ub 1024 | 48.7 | 766 | 1017 | OK | | ncmoe 32, ub 768 | 48.8 | 600 | 1273 | OK | | ncmoe 30, ub 1536 | 51.2 | 1009 | under | OOM | ## Chosen State Prefill speed is the priority, so the state keeps the top prefill tier and moves one expert layer to CPU for stability: ``` --n-cpu-moe 36 # 4 expert layers on GPU --batch-size 8192 --ubatch-size 3072 ``` This gives prefill about 1427 t/s at 16384 prompt tokens, generation about 44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool (2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and removes the context-proportional flash-attn F16-conversion workspace that caused the cuBLAS crash class). `ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with `ubatch 3072`, a big prefill burst failed a cuBLAS call under memory pressure (a `cudaErrorInvalidValue` in `ggml_cuda_mul_mat_cublas`), which killed the service. Raising `ncmoe` one step frees about 0.75 GiB of peak VRAM (the grid shows `ncmoe 36 - ubatch 3072` peak 8606 MiB versus 9360 MiB at `ncmoe 35`) for about 2 percent less prefill and 2 percent less decode. If a crash still occurs under load, set `GGML_CUDA_FORCE_MMQ=1` to bypass the cuBLAS path for quantized matmuls, or drop `ubatch`, or shrink the context pool. If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024` (generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom). ## Limitations llama-bench measures raw decode and prefill. It does not run speculative decoding, parallel slots, or the mmproj vision tower. The live service adds those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the real overhead varies a little with load. ## See Also - [[Dankserver System Specs]] - [[deployment]] - [[DFlash DSpark Speculative Decoding]]