frank-default-small: ncmoe 35->36 for prefill stability

Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.

ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
This commit is contained in:
Frank committed 2026-08-14 10:56:40 -07:00
1 parent 7daf9896e5
commit 9d7643ec5d
5 files changed
+28 -20

No files matched your search

+16 -9
View File
@@ -101,24 +101,31 @@ and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
## Chosen State
Prefill speed is the priority for this workload, so the final state takes
the top prefill tier while keeping a safe headroom margin:
Prefill speed is the priority, so the state keeps the top prefill tier and
moves one expert layer to CPU for stability:
```
--n-cpu-moe 35 # 5 expert layers on GPU
--n-cpu-moe 36 # 4 expert layers on GPU
--batch-size 8192
--ubatch-size 3072
```
This gives prefill about 1458 t/s at 16384 prompt tokens, generation about
45.4 t/s, and about 1.2 GiB of headroom at the 800000-token pool. Both are
chosen because generation does not depend on `ubatch`: `ubatch 3072` reaches
the best prefill speed with no cost to decode.
This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
44.4 t/s, and about 3.0 GiB of headroom at the 800000-token pool.
`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory
pressure (a `cudaErrorInvalidValue` in `ggml_cuda_mul_mat_cublas`), which
killed the service. Raising `ncmoe` one step frees about 0.75 GiB of peak
VRAM (the grid shows `ncmoe 36 - ubatch 3072` peak 8606 MiB versus 9360 MiB
at `ncmoe 35`) for about 2 percent less prefill and 2 percent less decode.
If a crash still occurs under load, set `GGML_CUDA_FORCE_MMQ=1` to bypass
the cuBLAS path for quantized matmuls, or drop `ubatch`, or shrink the
context pool.
If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
If stability headroom matters most, use `ncmoe 34 - ubatch 1536` (about 2.0
GiB headroom).
## Limitations