Frank
9d7643ec5d
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas killed the service during a 145k-token prefill burst at ncmoe35/ub3072. ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM (grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode. Keeps prefill priority (top ubatch tier). Docs synced; noted fallback GGML_CUDA_FORCE_MMQ=1 if pressure persists.
142 lines
5.6 KiB
Markdown
142 lines
5.6 KiB
Markdown
---
|
|
type: reference
|
|
title: frank-default-small Benchmark — ncmoe and Batch Tuning
|
|
description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
|
|
tags:
|
|
- benchmark
|
|
- llamacpp
|
|
- ncmoe
|
|
- ubatch
|
|
- dankserver
|
|
---
|
|
|
|
# frank-default-small Benchmark — ncmoe and Batch/Ubatch Tuning
|
|
|
|
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
|
|
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
|
|
`--n-cpu-moe` and `--ubatch-size` values for `frank-default-small`.
|
|
|
|
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
|
|
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
|
|
fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions.
|
|
|
|
## Headline Findings
|
|
|
|
### Generation does not depend on batch or ubatch
|
|
|
|
For every `ncmoe` value, generation throughput stayed flat across all
|
|
`ubatch` values from 256 to 4096. The delta between the lowest and highest
|
|
was under one token per second. Generation depends only on how many MoE
|
|
expert layers stay on the GPU as `ncmoe` gets smaller.
|
|
|
|
| `ncmoe` | Expert layers on GPU | Generation (t/s) |
|
|
| ------- | -------------------- | ---------------- |
|
|
| 40 | 0 | 40.9 |
|
|
| 38 | 2 | 42.6 |
|
|
| 36 | 4 | 44.4 |
|
|
| 35 | 5 | 45.5 |
|
|
| 34 | 6 | 46.4 |
|
|
| 32 | 8 | 48.7 |
|
|
| 30 | 10 | 51.1 |
|
|
|
|
Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of
|
|
generation. Moving from `ncmoe 35` to `ncmoe 32` adds about 3.2 t/s, or 7
|
|
percent.
|
|
|
|
### Prefill and graph memory scale with ubatch
|
|
|
|
Prefill throughput climbs with `ubatch` because a prompt splits into
|
|
`prompt / ubatch` micro-batches. For `ncmoe 35` at 16384 prompt tokens:
|
|
|
|
| `ubatch` | Prefill (t/s) | Peak graph VRAM (MiB) |
|
|
| -------- | ------------- | --------------------- |
|
|
| 256 | 245 | 6838 |
|
|
| 512 | 416 | 6852 |
|
|
| 768 | 556 | 7074 |
|
|
| 1024 | 710 | 7330 |
|
|
| 1536 | 895 | 7846 |
|
|
| 2048 | 1188 | 8574 |
|
|
|
|
The logical `--batch-size` does not matter. At a fixed `ubatch`, `batch`
|
|
values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM.
|
|
|
|
## Peak VRAM by ncmoe and ubatch
|
|
|
|
Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
|
|
(more experts on GPU) and when `ubatch` rises (bigger compute graph):
|
|
|
|
| `ncmoe` | ub 512 | ub 1024 | ub 1536 | ub 2048 |
|
|
| ------- | ------ | ------- | ------- | ------- |
|
|
| 38 | 4590 | 5068 | 5584 | 6086 |
|
|
| 36 | 6098 | 6576 | 7092 | 7594 |
|
|
| 35 | 6852 | 7330 | 7846 | 8574 |
|
|
| 34 | 7606 | 8084 | 8600 | 9102 |
|
|
| 32 | 9114 | 9592 | 10108 | 10610 |
|
|
| 30 | 10622 | 11100 | 11616 | 12118 |
|
|
|
|
## VRAM Budget Model
|
|
|
|
The live service holds a fixed 800000-token KV pool plus the mmproj vision
|
|
tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a
|
|
24 GiB budget:
|
|
|
|
```
|
|
service VRAM = llama-bench peak + KV pool (11.44 GiB) + overhead (2.2 GiB)
|
|
```
|
|
|
|
This model reproduces the historical OOMs. Configurations whose sum exceeds
|
|
24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about 24.4 GiB
|
|
and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
|
|
|
|
## Candidate Configurations
|
|
|
|
| Configuration | Gen (t/s) | Prefill (t/s) | Headroom (MiB) | Fit |
|
|
| ------------------ | --------- | ------------- | -------------- | --- |
|
|
| ncmoe 35, ub 2048 | 45.5 | 1188 | 2035 | OK |
|
|
| ncmoe 34, ub 2048 | 46.4 | 1221 | 1507 | OK |
|
|
| ncmoe 34, ub 1536 | 46.5 | 914 | 2009 | OK |
|
|
| ncmoe 32, ub 1024 | 48.7 | 766 | 1017 | OK |
|
|
| ncmoe 32, ub 768 | 48.8 | 600 | 1273 | OK |
|
|
| ncmoe 30, ub 1536 | 51.2 | 1009 | under | OOM |
|
|
|
|
## Chosen State
|
|
|
|
Prefill speed is the priority, so the state keeps the top prefill tier and
|
|
moves one expert layer to CPU for stability:
|
|
|
|
```
|
|
--n-cpu-moe 36 # 4 expert layers on GPU
|
|
--batch-size 8192
|
|
--ubatch-size 3072
|
|
```
|
|
|
|
This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
|
|
44.4 t/s, and about 3.0 GiB of headroom at the 800000-token pool.
|
|
|
|
`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
|
|
`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory
|
|
pressure (a `cudaErrorInvalidValue` in `ggml_cuda_mul_mat_cublas`), which
|
|
killed the service. Raising `ncmoe` one step frees about 0.75 GiB of peak
|
|
VRAM (the grid shows `ncmoe 36 - ubatch 3072` peak 8606 MiB versus 9360 MiB
|
|
at `ncmoe 35`) for about 2 percent less prefill and 2 percent less decode.
|
|
|
|
If a crash still occurs under load, set `GGML_CUDA_FORCE_MMQ=1` to bypass
|
|
the cuBLAS path for quantized matmuls, or drop `ubatch`, or shrink the
|
|
context pool.
|
|
|
|
If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
|
|
(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
|
|
|
|
## Limitations
|
|
|
|
llama-bench measures raw decode and prefill. It does not run speculative
|
|
decoding, parallel slots, or the mmproj vision tower. The live service adds
|
|
those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the
|
|
real overhead varies a little with load.
|
|
|
|
## See Also
|
|
|
|
- [[Dankserver System Specs]]
|
|
- [[deployment]]
|
|
- [[DFlash DSpark Speculative Decoding]]
|