frank-default-small: ncmoe 32 + ubatch 1024 from llama-bench sweep

Swept ncmoe(30..40) x ubatch(256..4096) on the RTX 3090 (Qwen3.6-35B-A3B Q6_K_P, pp16384/tg512, fit-target 0). Key findings: - Generation depends only on ncmoe: 40.9 (ncmoe40) .. 51.1 (ncmoe30); ubatch/batch do not affect decode at all. - Prefill + graph VRAM scale with ubatch; logical batch is irrelevant. - VRAM model (peak + 11.44GiB KV pool + 2.2GiB overhead) reproduces the historical OOMs: ncmoe32-ub2048 sums ~24.4GiB > 24. Chosen state: ncmoe 32 (8 expert layers on GPU), batch 8192, ubatch 1024. gen ~48.7 t/s (+7% vs prior ncmoe 35), prefill ~766 t/s, ~1.0GiB headroom. Alternates: ncmoe34-ub2048 (gen 46.4, pp 1221, 1.5GiB) for prefill-heavy; ncmoe32-ub768 (1.3GiB) if headroom matters most. Docs: new frank-default-small-benchmark.md; synced deployment.md + specs.
2026-08-13 22:56:24 -07:00
1 parent 0d09756430
commit 3c2db64a78
@@ -0,0 +1,129 @@
+---
+type: reference
+title: frank-default-small Benchmark — ncmoe and Batch Tuning
+description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
+tags:
+  - benchmark
+  - llamacpp
+  - ncmoe
+  - ubatch
+  - dankserver
+---
+
+# frank-default-small Benchmark — ncmoe and Batch/Ubatch Tuning
+
+This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
+the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
+`--n-cpu-moe` and `--ubatch-size` values for `frank-default-small`.
+
+Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
+`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
+fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions.
+
+## Headline Findings
+
+### Generation does not depend on batch or ubatch
+
+For every `ncmoe` value, generation throughput stayed flat across all
+`ubatch` values from 256 to 4096. The delta between the lowest and highest
+was under one token per second. Generation depends only on how many MoE
+expert layers stay on the GPU as `ncmoe` gets smaller.
+
+| `ncmoe` | Expert layers on GPU | Generation (t/s) |
+| ------- | -------------------- | ---------------- |
+| 40      | 0                    | 40.9             |
+| 38      | 2                    | 42.6             |
+| 36      | 4                    | 44.4             |
+| 35      | 5                    | 45.5             |
+| 34      | 6                    | 46.4             |
+| 32      | 8                    | 48.7             |
+| 30      | 10                   | 51.1             |
+
+Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of
+generation. Moving from `ncmoe 35` to `ncmoe 32` adds about 3.2 t/s, or 7
+percent.
+
+### Prefill and graph memory scale with ubatch
+
+Prefill throughput climbs with `ubatch` because a prompt splits into
+`prompt / ubatch` micro-batches. For `ncmoe 35` at 16384 prompt tokens:
+
+| `ubatch` | Prefill (t/s) | Peak graph VRAM (MiB) |
+| -------- | ------------- | --------------------- |
+| 256      | 245           | 6838                  |
+| 512      | 416           | 6852                  |
+| 768      | 556           | 7074                  |
+| 1024     | 710           | 7330                  |
+| 1536     | 895           | 7846                  |
+| 2048     | 1188          | 8574                  |
+
+The logical `--batch-size` does not matter. At a fixed `ubatch`, `batch`
+values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM.
+
+## Peak VRAM by ncmoe and ubatch
+
+Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
+(more experts on GPU) and when `ubatch` rises (bigger compute graph):
+
+| `ncmoe` | ub 512 | ub 1024 | ub 1536 | ub 2048 |
+| ------- | ------ | ------- | ------- | ------- |
+| 38      | 4590   | 5068    | 5584    | 6086    |
+| 36      | 6098   | 6576    | 7092    | 7594    |
+| 35      | 6852   | 7330    | 7846    | 8574    |
+| 34      | 7606   | 8084    | 8600    | 9102    |
+| 32      | 9114   | 9592    | 10108   | 10610   |
+| 30      | 10622  | 11100   | 11616   | 12118   |
+
+## VRAM Budget Model
+
+The live service holds a fixed 800000-token KV pool plus the mmproj vision
+tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a
+24 GiB budget:
+
+```
+service VRAM = llama-bench peak + KV pool (11.44 GiB) + overhead (2.2 GiB)
+```
+
+This model reproduces the historical OOMs. Configurations whose sum exceeds
+24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about 24.4 GiB
+and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
+
+## Candidate Configurations
+
+| Configuration      | Gen (t/s) | Prefill (t/s) | Headroom (MiB) | Fit |
+| ------------------ | --------- | ------------- | -------------- | --- |
+| ncmoe 35, ub 2048  | 45.5      | 1188          | 2035           | OK  |
+| ncmoe 34, ub 2048  | 46.4      | 1221          | 1507           | OK  |
+| ncmoe 34, ub 1536  | 46.5      | 914           | 2009           | OK  |
+| ncmoe 32, ub 1024  | 48.7      | 766           | 1017           | OK  |
+| ncmoe 32, ub 768   | 48.8      | 600           | 1273           | OK  |
+| ncmoe 30, ub 1536  | 51.2      | 1009          | under          | OOM |
+
+## Chosen State
+
+The chosen state balances generation speed against prefill speed and OOM
+headroom:
+
+```
+--n-cpu-moe 32     # 8 expert layers on GPU
+--batch-size   8192
+--ubatch-size  1024
+```
+
+This gives generation about 48.7 t/s (7 percent above the previous
+`ncmoe 35`) while keeping the box above the OOM boundary. If prefill speed
+matters more than generation, use `ncmoe 34 - ubatch 2048`. If stability
+headroom matters most, use `ncmoe 34 - ubatch 1536`.
+
+## Limitations
+
+llama-bench measures raw decode and prefill. It does not run speculative
+decoding, parallel slots, or the mmproj vision tower. The live service adds
+those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the
+real overhead varies a little with load.
+
+## See Also
+
+- [[Dankserver System Specs]]
+- [[deployment]]
+- [[DFlash DSpark Speculative Decoding]]