dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention

Browse Source 
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
This commit is contained in:
Frank committed 2026-08-28 02:20:06 -07:00
1 parent 6d4a96a423
commit 5829a0bfe8
140 files changed
+1820 -86
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/frank-default-small-benchmark.md
+181
View File
@@ -0,0 +1,181 @@
---
type: reference
title: frank-default-small Benchmark — ncmoe and Batch Tuning
description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
tags:
- benchmark
- llamacpp
- ncmoe
- ubatch
- dankserver
---
# frank-default-small Benchmark — ncmoe and Batch/Ubatch Tuning
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
`--n-cpu-moe` and `--ubatch-size` values for `frank-default-small`.
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions.
## Headline Findings
### Generation does not depend on batch or ubatch
For every `ncmoe` value, generation throughput stayed flat across all
`ubatch` values from 256 to 4096. The delta between the lowest and highest
was under one token per second. Generation depends only on how many MoE
expert layers stay on the GPU as `ncmoe` gets smaller.
| `ncmoe` | Expert layers on GPU | Generation (t/s) |
| ------- | -------------------- | ---------------- |
| 40 | 0 | 40.9 |
| 38 | 2 | 42.6 |
| 36 | 4 | 44.4 |
| 35 | 5 | 45.5 |
| 34 | 6 | 46.4 |
| 32 | 8 | 48.7 |
| 30 | 10 | 51.1 |
Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of
generation. Moving from `ncmoe 35` to `ncmoe 32` adds about 3.2 t/s, or 7
percent.
### Prefill and graph memory scale with ubatch
Prefill throughput climbs with `ubatch` because a prompt splits into
`prompt / ubatch` micro-batches. For `ncmoe 35` at 16384 prompt tokens:
| `ubatch` | Prefill (t/s) | Peak graph VRAM (MiB) |
| -------- | ------------- | --------------------- |
| 256 | 245 | 6838 |
| 512 | 416 | 6852 |
| 768 | 556 | 7074 |
| 1024 | 710 | 7330 |
| 1536 | 895 | 7846 |
| 2048 | 1188 | 8574 |
The logical `--batch-size` does not matter. At a fixed `ubatch`, `batch`
values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM.
## Peak VRAM by ncmoe and ubatch
Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
(more experts on GPU) and when `ubatch` rises (bigger compute graph):
| `ncmoe` | ub 512 | ub 1024 | ub 1536 | ub 2048 |
| ------- | ------ | ------- | ------- | ------- |
| 38 | 4590 | 5068 | 5584 | 6086 |
| 36 | 6098 | 6576 | 7092 | 7594 |
| 35 | 6852 | 7330 | 7846 | 8574 |
| 34 | 7606 | 8084 | 8600 | 9102 |
| 32 | 9114 | 9592 | 10108 | 10610 |
| 30 | 10622 | 11100 | 11616 | 12118 |
## VRAM Budget Model
The live service holds a fixed 700416-token KV pool (f16/f16 since
2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA
overhead. The model below fits llama-bench peak VRAM into a 24 GiB
budget:
```
service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB)
```
This model reproduces the historical OOMs. Configurations whose sum
exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about
24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and
`33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at
800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the
service runs 700416 cells (13,680 MiB).
## Candidate Configurations
| Configuration | Gen (t/s) | Prefill (t/s) | Headroom (MiB) | Fit |
| ------------------ | --------- | ------------- | -------------- | --- |
| ncmoe 35, ub 2048 | 45.5 | 1188 | 2035 | OK |
| ncmoe 34, ub 2048 | 46.4 | 1221 | 1507 | OK |
| ncmoe 34, ub 1536 | 46.5 | 914 | 2009 | OK |
| ncmoe 32, ub 1024 | 48.7 | 766 | 1017 | OK |
| ncmoe 32, ub 768 | 48.8 | 600 | 1273 | OK |
| ncmoe 30, ub 1536 | 51.2 | 1009 | under | OOM |
## Chosen State
> **Live drift (2026-08-22 audit):** the deployed `frank-default-small` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
Prefill speed is the priority, so the state keeps the top prefill tier and
moves one expert layer to CPU for stability:
```
--n-cpu-moe 36 # 4 expert layers on GPU
--batch-size 8192
--ubatch-size 3072
```
This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool
(2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and
removes the context-proportional flash-attn F16-conversion workspace that
caused the cuBLAS crash class).
`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory
pressure (a `cudaErrorInvalidValue` in `ggml_cuda_mul_mat_cublas`), which
killed the service. Raising `ncmoe` one step frees about 0.75 GiB of peak
VRAM (the grid shows `ncmoe 36 - ubatch 3072` peak 8606 MiB versus 9360 MiB
at `ncmoe 35`) for about 2 percent less prefill and 2 percent less decode.
If a crash still occurs under load, set `GGML_CUDA_FORCE_MMQ=1` to bypass
the cuBLAS path for quantized matmuls, or drop `ubatch`, or shrink the
context pool.
If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
## Ornith Q6_K — 8K `ubatch` Sweep Without Speculation
Run date: 2026-08-25. This controlled sweep used the active Ornith Q6_K model
and hybrid expert placement, but explicitly disabled speculative decoding with
`--spec-type none`. It used one slot, an 8192-token context, a fixed logical
`--batch-size 4096`, prompt caching disabled, and one unique approximately
8K-token prompt per point. The production service was stopped during the
sweep and automatically restarted afterward.
| `ubatch` | Prompt tokens/s | Generation tokens/s | CUDA compute reservation |
|---:|---:|---:|---:|
| 1024 | 1944 | 53.91 | 248 MiB |
| 1280 | 2030 | 53.87 | 248 MiB |
| 1536 | 2190 | 53.66 | 248 MiB |
| 1792 | 2203 | 53.69 | 248 MiB |
| 2048 | 2423 | 53.80 | 248 MiB |
Prefill improves through `ubatch=2048`, while generation is effectively flat.
The 8K prompt sweep therefore favors `ubatch=2048` as the balanced point; a
larger value was not tested in this corrected no-spec run.
The same workload with `draft-mtp` enabled produced approximately 26.8–35.2
generation tokens/s at these `ubatch` values, with variable draft acceptance.
The no-spec path produced approximately 53.7–53.9 generation tokens/s. This
means MTP was a net throughput loss for this workload and configuration,
despite accepting draft tokens. The production service remains on MTP until a
separate decision is made; acceptance count alone is not sufficient to select
speculation.
## Limitations
llama-bench measures raw decode and prefill. It does not run speculative
decoding, parallel slots, or the mmproj vision tower. The live service adds
those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the
real overhead varies a little with load. The Ornith sweep above used the
OpenAI-compatible server API rather than `llama-bench`, and its throughput
includes the server's timing instrumentation.
## See Also
- [[Dankserver System Specs]]
- [[deployment]]
- [[DFlash DSpark Speculative Decoding]]