Frank
e7f0d93c86
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
194 lines
8.5 KiB
Markdown
194 lines
8.5 KiB
Markdown
---
|
||
timestamp: 2026-09-03
|
||
type: reference
|
||
title: frank-zealot Benchmark — ncmoe and Batch Tuning
|
||
description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
|
||
tags:
|
||
- benchmark
|
||
- llamacpp
|
||
- ncmoe
|
||
- ubatch
|
||
- dankserver
|
||
---
|
||
|
||
# frank-zealot Benchmark — ncmoe and Batch/Ubatch Tuning
|
||
|
||
> **HISTORY (runs 2026-08-13 and 2026-08-25, Dankserver's RTX 3090).** That
|
||
> card was pulled from Dankserver on 2026-09-15 (`lspci | grep -ci nvidia` → 0
|
||
> there on 2026-10-18 — [[Dankserver System Specs]]) and today drives
|
||
> frank-zealot on **Dankclaw**: [[frank-zealot-llama-prism-bonsai2-pq20]]
|
||
> (`Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off, no
|
||
> `--n-cpu-moe` — the model fits the card outright). What survived the payloads:
|
||
> decode is ubatch-invariant, CUDA prefill climbs monotonically to ubatch 2048,
|
||
> and the logical `--batch-size` is neutral — the live env still ships
|
||
> 4096/2048 (`scripts/frank-zealot.env:44-45`).
|
||
|
||
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
|
||
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
|
||
`--n-cpu-moe` and `--ubatch-size` values for the llama.cpp rung then served on
|
||
that card (the `frank-zealot` name dates from 2026-10-06/10-09).
|
||
|
||
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
|
||
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
|
||
fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions.
|
||
|
||
## Headline Findings
|
||
|
||
### Generation does not depend on batch or ubatch
|
||
|
||
For every `ncmoe` value, generation throughput stayed flat across all
|
||
`ubatch` values from 256 to 4096. The delta between the lowest and highest
|
||
was under one token per second. Generation depends only on how many MoE
|
||
expert layers stay on the GPU as `ncmoe` gets smaller.
|
||
|
||
| `ncmoe` | Expert layers on GPU | Generation (t/s) |
|
||
| ------- | -------------------- | ---------------- |
|
||
| 40 | 0 | 40.9 |
|
||
| 38 | 2 | 42.6 |
|
||
| 36 | 4 | 44.4 |
|
||
| 35 | 5 | 45.5 |
|
||
| 34 | 6 | 46.4 |
|
||
| 32 | 8 | 48.7 |
|
||
| 30 | 10 | 51.1 |
|
||
|
||
Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of
|
||
generation. Moving from `ncmoe 35` to `ncmoe 32` adds about 3.2 t/s, or 7
|
||
percent.
|
||
|
||
### Prefill and graph memory scale with ubatch
|
||
|
||
Prefill throughput climbs with `ubatch` because a prompt splits into
|
||
`prompt / ubatch` micro-batches. For `ncmoe 35` at 16384 prompt tokens:
|
||
|
||
| `ubatch` | Prefill (t/s) | Peak graph VRAM (MiB) |
|
||
| -------- | ------------- | --------------------- |
|
||
| 256 | 245 | 6838 |
|
||
| 512 | 416 | 6852 |
|
||
| 768 | 556 | 7074 |
|
||
| 1024 | 710 | 7330 |
|
||
| 1536 | 895 | 7846 |
|
||
| 2048 | 1188 | 8574 |
|
||
|
||
The logical `--batch-size` does not matter. At a fixed `ubatch`, `batch`
|
||
values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM.
|
||
|
||
## Peak VRAM by ncmoe and ubatch
|
||
|
||
Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
|
||
(more experts on GPU) and when `ubatch` rises (bigger compute graph):
|
||
|
||
| `ncmoe` | ub 512 | ub 1024 | ub 1536 | ub 2048 |
|
||
| ------- | ------ | ------- | ------- | ------- |
|
||
| 38 | 4590 | 5068 | 5584 | 6086 |
|
||
| 36 | 6098 | 6576 | 7092 | 7594 |
|
||
| 35 | 6852 | 7330 | 7846 | 8574 |
|
||
| 34 | 7606 | 8084 | 8600 | 9102 |
|
||
| 32 | 9114 | 9592 | 10108 | 10610 |
|
||
| 30 | 10622 | 11100 | 11616 | 12118 |
|
||
|
||
## VRAM Budget Model
|
||
|
||
The live service holds a fixed 700416-token KV pool (f16/f16 since
|
||
2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA
|
||
overhead. The model below fits llama-bench peak VRAM into a 24 GiB
|
||
budget:
|
||
|
||
```
|
||
service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB)
|
||
```
|
||
|
||
This model reproduces the historical OOMs. Configurations whose sum
|
||
exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about
|
||
24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and
|
||
`33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at
|
||
800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the
|
||
service runs 700416 cells (13,680 MiB).
|
||
|
||
## Candidate Configurations
|
||
|
||
| Configuration | Gen (t/s) | Prefill (t/s) | Headroom (MiB) | Fit |
|
||
| ------------------ | --------- | ------------- | -------------- | --- |
|
||
| ncmoe 35, ub 2048 | 45.5 | 1188 | 2035 | OK |
|
||
| ncmoe 34, ub 2048 | 46.4 | 1221 | 1507 | OK |
|
||
| ncmoe 34, ub 1536 | 46.5 | 914 | 2009 | OK |
|
||
| ncmoe 32, ub 1024 | 48.7 | 766 | 1017 | OK |
|
||
| ncmoe 32, ub 768 | 48.8 | 600 | 1273 | OK |
|
||
| ncmoe 30, ub 1536 | 51.2 | 1009 | under | OOM |
|
||
|
||
## Chosen State
|
||
|
||
> **Live drift (2026-08-22 audit):** the deployed `frank-zealot` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
|
||
|
||
Prefill speed is the priority, so the state keeps the top prefill tier and
|
||
moves one expert layer to CPU for stability:
|
||
|
||
```
|
||
--n-cpu-moe 36 # 4 expert layers on GPU
|
||
--batch-size 8192
|
||
--ubatch-size 3072
|
||
```
|
||
|
||
This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
|
||
44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool
|
||
(2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and
|
||
removes the context-proportional flash-attn F16-conversion workspace that
|
||
caused the cuBLAS crash class).
|
||
|
||
`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
|
||
`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory
|
||
pressure (a `cudaErrorInvalidValue` in `ggml_cuda_mul_mat_cublas`), which
|
||
killed the service. Raising `ncmoe` one step frees about 0.75 GiB of peak
|
||
VRAM (the grid shows `ncmoe 36 - ubatch 3072` peak 8606 MiB versus 9360 MiB
|
||
at `ncmoe 35`) for about 2 percent less prefill and 2 percent less decode.
|
||
|
||
If a crash still occurs under load, set `GGML_CUDA_FORCE_MMQ=1` to bypass
|
||
the cuBLAS path for quantized matmuls, or drop `ubatch`, or shrink the
|
||
context pool.
|
||
|
||
If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
|
||
(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
|
||
|
||
## Ornith Q6_K — 8K `ubatch` Sweep Without Speculation
|
||
|
||
Run date: 2026-08-25. This controlled sweep used the active Ornith Q6_K model
|
||
and hybrid expert placement, but explicitly disabled speculative decoding with
|
||
`--spec-type none`. It used one slot, an 8192-token context, a fixed logical
|
||
`--batch-size 4096`, prompt caching disabled, and one unique approximately
|
||
8K-token prompt per point. The production service was stopped during the
|
||
sweep and automatically restarted afterward.
|
||
|
||
| `ubatch` | Prompt tokens/s | Generation tokens/s | CUDA compute reservation |
|
||
|---:|---:|---:|---:|
|
||
| 1024 | 1944 | 53.91 | 248 MiB |
|
||
| 1280 | 2030 | 53.87 | 248 MiB |
|
||
| 1536 | 2190 | 53.66 | 248 MiB |
|
||
| 1792 | 2203 | 53.69 | 248 MiB |
|
||
| 2048 | 2423 | 53.80 | 248 MiB |
|
||
|
||
Prefill improves through `ubatch=2048`, while generation is effectively flat.
|
||
The 8K prompt sweep therefore favors `ubatch=2048` as the balanced point; a
|
||
larger value was not tested in this corrected no-spec run.
|
||
|
||
The same workload with `draft-mtp` enabled produced approximately 26.8–35.2
|
||
generation tokens/s at these `ubatch` values, with variable draft acceptance.
|
||
The no-spec path produced approximately 53.7–53.9 generation tokens/s. This
|
||
means MTP was a net throughput loss for this workload and configuration,
|
||
despite accepting draft tokens. The production service remains on MTP until a
|
||
separate decision is made; acceptance count alone is not sufficient to select
|
||
speculation.
|
||
|
||
## Limitations
|
||
|
||
llama-bench measures raw decode and prefill. It does not run speculative
|
||
decoding, parallel slots, or the mmproj vision tower. The live service adds
|
||
those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the
|
||
real overhead varies a little with load. The Ornith sweep above used the
|
||
OpenAI-compatible server API rather than `llama-bench`, and its throughput
|
||
includes the server's timing instrumentation.
|
||
|
||
## See Also
|
||
|
||
- [[Dankserver System Specs]]
|
||
- [[deployment]]
|
||
- [[DFlash DSpark Speculative Decoding]]
|