Frank
01eb957f90
Prefill is the priority for this workload. Switch from decode-max (ncmoe 32 / ubatch 1024) to prefill-max: - n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s, ~1.2 GiB headroom at the 800k pool. - Gen is independent of ubatch; ub 3072 reaches best prefill with no decode cost. Docs synced (benchmark Chosen State, deployment, specs).
135 lines
5.3 KiB
Markdown
135 lines
5.3 KiB
Markdown
---
|
|
type: reference
|
|
title: frank-default-small Benchmark — ncmoe and Batch Tuning
|
|
description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
|
|
tags:
|
|
- benchmark
|
|
- llamacpp
|
|
- ncmoe
|
|
- ubatch
|
|
- dankserver
|
|
---
|
|
|
|
# frank-default-small Benchmark — ncmoe and Batch/Ubatch Tuning
|
|
|
|
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
|
|
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
|
|
`--n-cpu-moe` and `--ubatch-size` values for `frank-default-small`.
|
|
|
|
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
|
|
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
|
|
fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions.
|
|
|
|
## Headline Findings
|
|
|
|
### Generation does not depend on batch or ubatch
|
|
|
|
For every `ncmoe` value, generation throughput stayed flat across all
|
|
`ubatch` values from 256 to 4096. The delta between the lowest and highest
|
|
was under one token per second. Generation depends only on how many MoE
|
|
expert layers stay on the GPU as `ncmoe` gets smaller.
|
|
|
|
| `ncmoe` | Expert layers on GPU | Generation (t/s) |
|
|
| ------- | -------------------- | ---------------- |
|
|
| 40 | 0 | 40.9 |
|
|
| 38 | 2 | 42.6 |
|
|
| 36 | 4 | 44.4 |
|
|
| 35 | 5 | 45.5 |
|
|
| 34 | 6 | 46.4 |
|
|
| 32 | 8 | 48.7 |
|
|
| 30 | 10 | 51.1 |
|
|
|
|
Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of
|
|
generation. Moving from `ncmoe 35` to `ncmoe 32` adds about 3.2 t/s, or 7
|
|
percent.
|
|
|
|
### Prefill and graph memory scale with ubatch
|
|
|
|
Prefill throughput climbs with `ubatch` because a prompt splits into
|
|
`prompt / ubatch` micro-batches. For `ncmoe 35` at 16384 prompt tokens:
|
|
|
|
| `ubatch` | Prefill (t/s) | Peak graph VRAM (MiB) |
|
|
| -------- | ------------- | --------------------- |
|
|
| 256 | 245 | 6838 |
|
|
| 512 | 416 | 6852 |
|
|
| 768 | 556 | 7074 |
|
|
| 1024 | 710 | 7330 |
|
|
| 1536 | 895 | 7846 |
|
|
| 2048 | 1188 | 8574 |
|
|
|
|
The logical `--batch-size` does not matter. At a fixed `ubatch`, `batch`
|
|
values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM.
|
|
|
|
## Peak VRAM by ncmoe and ubatch
|
|
|
|
Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
|
|
(more experts on GPU) and when `ubatch` rises (bigger compute graph):
|
|
|
|
| `ncmoe` | ub 512 | ub 1024 | ub 1536 | ub 2048 |
|
|
| ------- | ------ | ------- | ------- | ------- |
|
|
| 38 | 4590 | 5068 | 5584 | 6086 |
|
|
| 36 | 6098 | 6576 | 7092 | 7594 |
|
|
| 35 | 6852 | 7330 | 7846 | 8574 |
|
|
| 34 | 7606 | 8084 | 8600 | 9102 |
|
|
| 32 | 9114 | 9592 | 10108 | 10610 |
|
|
| 30 | 10622 | 11100 | 11616 | 12118 |
|
|
|
|
## VRAM Budget Model
|
|
|
|
The live service holds a fixed 800000-token KV pool plus the mmproj vision
|
|
tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a
|
|
24 GiB budget:
|
|
|
|
```
|
|
service VRAM = llama-bench peak + KV pool (11.44 GiB) + overhead (2.2 GiB)
|
|
```
|
|
|
|
This model reproduces the historical OOMs. Configurations whose sum exceeds
|
|
24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about 24.4 GiB
|
|
and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
|
|
|
|
## Candidate Configurations
|
|
|
|
| Configuration | Gen (t/s) | Prefill (t/s) | Headroom (MiB) | Fit |
|
|
| ------------------ | --------- | ------------- | -------------- | --- |
|
|
| ncmoe 35, ub 2048 | 45.5 | 1188 | 2035 | OK |
|
|
| ncmoe 34, ub 2048 | 46.4 | 1221 | 1507 | OK |
|
|
| ncmoe 34, ub 1536 | 46.5 | 914 | 2009 | OK |
|
|
| ncmoe 32, ub 1024 | 48.7 | 766 | 1017 | OK |
|
|
| ncmoe 32, ub 768 | 48.8 | 600 | 1273 | OK |
|
|
| ncmoe 30, ub 1536 | 51.2 | 1009 | under | OOM |
|
|
|
|
## Chosen State
|
|
|
|
Prefill speed is the priority for this workload, so the final state takes
|
|
the top prefill tier while keeping a safe headroom margin:
|
|
|
|
```
|
|
--n-cpu-moe 35 # 5 expert layers on GPU
|
|
--batch-size 8192
|
|
--ubatch-size 3072
|
|
```
|
|
|
|
This gives prefill about 1458 t/s at 16384 prompt tokens, generation about
|
|
45.4 t/s, and about 1.2 GiB of headroom at the 800000-token pool. Both are
|
|
chosen because generation does not depend on `ubatch`: `ubatch 3072` reaches
|
|
the best prefill speed with no cost to decode.
|
|
|
|
If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
|
|
(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
|
|
If stability headroom matters most, use `ncmoe 34 - ubatch 1536` (about 2.0
|
|
GiB headroom).
|
|
|
|
## Limitations
|
|
|
|
llama-bench measures raw decode and prefill. It does not run speculative
|
|
decoding, parallel slots, or the mmproj vision tower. The live service adds
|
|
those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the
|
|
real overhead varies a little with load.
|
|
|
|
## See Also
|
|
|
|
- [[Dankserver System Specs]]
|
|
- [[deployment]]
|
|
- [[DFlash DSpark Speculative Decoding]]
|