--- timestamp: 2026-09-03 type: reference title: frank-zealot Benchmark — ncmoe and Batch Tuning description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P. tags: - benchmark - llamacpp - ncmoe - ubatch - dankserver --- # frank-zealot Benchmark — ncmoe and Batch/Ubatch Tuning > **HISTORY (runs 2026-08-13 and 2026-08-25, Dankserver's RTX 3090).** That > card was pulled from Dankserver on 2026-09-15 (`lspci | grep -ci nvidia` → 0 > there on 2026-10-18 — [[Dankserver System Specs]]) and today drives > frank-zealot on **Dankclaw**: [[frank-zealot-llama-prism-bonsai2-pq20]] > (`Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off, no > `--n-cpu-moe` — the model fits the card outright). What survived the payloads: > decode is ubatch-invariant, CUDA prefill climbs monotonically to ubatch 2048, > and the logical `--batch-size` is neutral — the live env still ships > 4096/2048 (`scripts/frank-zealot.env:44-45`). This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final `--n-cpu-moe` and `--ubatch-size` values for the llama.cpp rung then served on that card (the `frank-zealot` name dates from 2026-10-06/10-09). Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads, `--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions. ## Headline Findings ### Generation does not depend on batch or ubatch For every `ncmoe` value, generation throughput stayed flat across all `ubatch` values from 256 to 4096. The delta between the lowest and highest was under one token per second. Generation depends only on how many MoE expert layers stay on the GPU as `ncmoe` gets smaller. | `ncmoe` | Expert layers on GPU | Generation (t/s) | | ------- | -------------------- | ---------------- | | 40 | 0 | 40.9 | | 38 | 2 | 42.6 | | 36 | 4 | 44.4 | | 35 | 5 | 45.5 | | 34 | 6 | 46.4 | | 32 | 8 | 48.7 | | 30 | 10 | 51.1 | Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of generation. Moving from `ncmoe 35` to `ncmoe 32` adds about 3.2 t/s, or 7 percent. ### Prefill and graph memory scale with ubatch Prefill throughput climbs with `ubatch` because a prompt splits into `prompt / ubatch` micro-batches. For `ncmoe 35` at 16384 prompt tokens: | `ubatch` | Prefill (t/s) | Peak graph VRAM (MiB) | | -------- | ------------- | --------------------- | | 256 | 245 | 6838 | | 512 | 416 | 6852 | | 768 | 556 | 7074 | | 1024 | 710 | 7330 | | 1536 | 895 | 7846 | | 2048 | 1188 | 8574 | The logical `--batch-size` does not matter. At a fixed `ubatch`, `batch` values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM. ## Peak VRAM by ncmoe and ubatch Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops (more experts on GPU) and when `ubatch` rises (bigger compute graph): | `ncmoe` | ub 512 | ub 1024 | ub 1536 | ub 2048 | | ------- | ------ | ------- | ------- | ------- | | 38 | 4590 | 5068 | 5584 | 6086 | | 36 | 6098 | 6576 | 7092 | 7594 | | 35 | 6852 | 7330 | 7846 | 8574 | | 34 | 7606 | 8084 | 8600 | 9102 | | 32 | 9114 | 9592 | 10108 | 10610 | | 30 | 10622 | 11100 | 11616 | 12118 | ## VRAM Budget Model The live service holds a fixed 700416-token KV pool (f16/f16 since 2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a 24 GiB budget: ``` service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB) ``` This model reproduces the historical OOMs. Configurations whose sum exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about 24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and `33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at 800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the service runs 700416 cells (13,680 MiB). ## Candidate Configurations | Configuration | Gen (t/s) | Prefill (t/s) | Headroom (MiB) | Fit | | ------------------ | --------- | ------------- | -------------- | --- | | ncmoe 35, ub 2048 | 45.5 | 1188 | 2035 | OK | | ncmoe 34, ub 2048 | 46.4 | 1221 | 1507 | OK | | ncmoe 34, ub 1536 | 46.5 | 914 | 2009 | OK | | ncmoe 32, ub 1024 | 48.7 | 766 | 1017 | OK | | ncmoe 32, ub 768 | 48.8 | 600 | 1273 | OK | | ncmoe 30, ub 1536 | 51.2 | 1009 | under | OOM | ## Chosen State > **Live drift (2026-08-22 audit):** the deployed `frank-zealot` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation. Prefill speed is the priority, so the state keeps the top prefill tier and moves one expert layer to CPU for stability: ``` --n-cpu-moe 36 # 4 expert layers on GPU --batch-size 8192 --ubatch-size 3072 ``` This gives prefill about 1427 t/s at 16384 prompt tokens, generation about 44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool (2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and removes the context-proportional flash-attn F16-conversion workspace that caused the cuBLAS crash class). `ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with `ubatch 3072`, a big prefill burst failed a cuBLAS call under memory pressure (a `cudaErrorInvalidValue` in `ggml_cuda_mul_mat_cublas`), which killed the service. Raising `ncmoe` one step frees about 0.75 GiB of peak VRAM (the grid shows `ncmoe 36 - ubatch 3072` peak 8606 MiB versus 9360 MiB at `ncmoe 35`) for about 2 percent less prefill and 2 percent less decode. If a crash still occurs under load, set `GGML_CUDA_FORCE_MMQ=1` to bypass the cuBLAS path for quantized matmuls, or drop `ubatch`, or shrink the context pool. If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024` (generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom). ## Ornith Q6_K — 8K `ubatch` Sweep Without Speculation Run date: 2026-08-25. This controlled sweep used the active Ornith Q6_K model and hybrid expert placement, but explicitly disabled speculative decoding with `--spec-type none`. It used one slot, an 8192-token context, a fixed logical `--batch-size 4096`, prompt caching disabled, and one unique approximately 8K-token prompt per point. The production service was stopped during the sweep and automatically restarted afterward. | `ubatch` | Prompt tokens/s | Generation tokens/s | CUDA compute reservation | |---:|---:|---:|---:| | 1024 | 1944 | 53.91 | 248 MiB | | 1280 | 2030 | 53.87 | 248 MiB | | 1536 | 2190 | 53.66 | 248 MiB | | 1792 | 2203 | 53.69 | 248 MiB | | 2048 | 2423 | 53.80 | 248 MiB | Prefill improves through `ubatch=2048`, while generation is effectively flat. The 8K prompt sweep therefore favors `ubatch=2048` as the balanced point; a larger value was not tested in this corrected no-spec run. The same workload with `draft-mtp` enabled produced approximately 26.8–35.2 generation tokens/s at these `ubatch` values, with variable draft acceptance. The no-spec path produced approximately 53.7–53.9 generation tokens/s. This means MTP was a net throughput loss for this workload and configuration, despite accepting draft tokens. The production service remains on MTP until a separate decision is made; acceptance count alone is not sufficient to select speculation. ## Limitations llama-bench measures raw decode and prefill. It does not run speculative decoding, parallel slots, or the mmproj vision tower. The live service adds those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the real overhead varies a little with load. The Ornith sweep above used the OpenAI-compatible server API rather than `llama-bench`, and its throughput includes the server's timing instrumentation. ## See Also - [[Dankserver System Specs]] - [[deployment]] - [[DFlash DSpark Speculative Decoding]]