frank-default-small: ncmoe 35->36 for prefill stability
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas killed the service during a 145k-token prefill burst at ncmoe35/ub3072. ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM (grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode. Keeps prefill priority (top ubatch tier). Docs synced; noted fallback GGML_CUDA_FORCE_MMQ=1 if pressure persists.
This commit is contained in:
1 parent
7daf9896e5
commit
9d7643ec5d
5 files changed
+28
-20
No files matched your search
@@ -101,24 +101,31 @@ and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
|
||||
|
||||
## Chosen State
|
||||
|
||||
Prefill speed is the priority for this workload, so the final state takes
|
||||
the top prefill tier while keeping a safe headroom margin:
|
||||
Prefill speed is the priority, so the state keeps the top prefill tier and
|
||||
moves one expert layer to CPU for stability:
|
||||
|
||||
```
|
||||
--n-cpu-moe 35 # 5 expert layers on GPU
|
||||
--n-cpu-moe 36 # 4 expert layers on GPU
|
||||
--batch-size 8192
|
||||
--ubatch-size 3072
|
||||
```
|
||||
|
||||
This gives prefill about 1458 t/s at 16384 prompt tokens, generation about
|
||||
45.4 t/s, and about 1.2 GiB of headroom at the 800000-token pool. Both are
|
||||
chosen because generation does not depend on `ubatch`: `ubatch 3072` reaches
|
||||
the best prefill speed with no cost to decode.
|
||||
This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
|
||||
44.4 t/s, and about 3.0 GiB of headroom at the 800000-token pool.
|
||||
|
||||
`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
|
||||
`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory
|
||||
pressure (a `cudaErrorInvalidValue` in `ggml_cuda_mul_mat_cublas`), which
|
||||
killed the service. Raising `ncmoe` one step frees about 0.75 GiB of peak
|
||||
VRAM (the grid shows `ncmoe 36 - ubatch 3072` peak 8606 MiB versus 9360 MiB
|
||||
at `ncmoe 35`) for about 2 percent less prefill and 2 percent less decode.
|
||||
|
||||
If a crash still occurs under load, set `GGML_CUDA_FORCE_MMQ=1` to bypass
|
||||
the cuBLAS path for quantized matmuls, or drop `ubatch`, or shrink the
|
||||
context pool.
|
||||
|
||||
If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
|
||||
(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
|
||||
If stability headroom matters most, use `ncmoe 34 - ubatch 1536` (about 2.0
|
||||
GiB headroom).
|
||||
|
||||
## Limitations
|
||||
|
||||
|
||||
Reference in new issue
Block a user