frank-default-small: prefill-max ncmoe 35 / ubatch 3072

Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
  ~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
  decode cost. Docs synced (benchmark Chosen State, deployment, specs).
This commit is contained in:
Frank committed 2026-08-13 23:07:30 -07:00
1 parent 2b4e58a926
commit 01eb957f90
5 files changed
+28 -23

No files matched your search

+13 -8
View File
@@ -101,19 +101,24 @@ and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
## Chosen State
The chosen state balances generation speed against prefill speed and OOM
headroom:
Prefill speed is the priority for this workload, so the final state takes
the top prefill tier while keeping a safe headroom margin:
```
--n-cpu-moe 32 # 8 expert layers on GPU
--n-cpu-moe 35 # 5 expert layers on GPU
--batch-size 8192
--ubatch-size 1024
--ubatch-size 3072
```
This gives generation about 48.7 t/s (7 percent above the previous
`ncmoe 35`) while keeping the box above the OOM boundary. If prefill speed
matters more than generation, use `ncmoe 34 - ubatch 2048`. If stability
headroom matters most, use `ncmoe 34 - ubatch 1536`.
This gives prefill about 1458 t/s at 16384 prompt tokens, generation about
45.4 t/s, and about 1.2 GiB of headroom at the 800000-token pool. Both are
chosen because generation does not depend on `ubatch`: `ubatch 3072` reaches
the best prefill speed with no cost to decode.
If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
If stability headroom matters most, use `ncmoe 34 - ubatch 1536` (about 2.0
GiB headroom).
## Limitations