frank-default-small: prefill-max ncmoe 35 / ubatch 3072
Prefill is the priority for this workload. Switch from decode-max (ncmoe 32 / ubatch 1024) to prefill-max: - n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s, ~1.2 GiB headroom at the 800k pool. - Gen is independent of ubatch; ub 3072 reaches best prefill with no decode cost. Docs synced (benchmark Chosen State, deployment, specs).
This commit is contained in:
1 parent
2b4e58a926
commit
01eb957f90
5 files changed
+28
-23
No files matched your search
@@ -101,19 +101,24 @@ and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
|
||||
|
||||
## Chosen State
|
||||
|
||||
The chosen state balances generation speed against prefill speed and OOM
|
||||
headroom:
|
||||
Prefill speed is the priority for this workload, so the final state takes
|
||||
the top prefill tier while keeping a safe headroom margin:
|
||||
|
||||
```
|
||||
--n-cpu-moe 32 # 8 expert layers on GPU
|
||||
--n-cpu-moe 35 # 5 expert layers on GPU
|
||||
--batch-size 8192
|
||||
--ubatch-size 1024
|
||||
--ubatch-size 3072
|
||||
```
|
||||
|
||||
This gives generation about 48.7 t/s (7 percent above the previous
|
||||
`ncmoe 35`) while keeping the box above the OOM boundary. If prefill speed
|
||||
matters more than generation, use `ncmoe 34 - ubatch 2048`. If stability
|
||||
headroom matters most, use `ncmoe 34 - ubatch 1536`.
|
||||
This gives prefill about 1458 t/s at 16384 prompt tokens, generation about
|
||||
45.4 t/s, and about 1.2 GiB of headroom at the 800000-token pool. Both are
|
||||
chosen because generation does not depend on `ubatch`: `ubatch 3072` reaches
|
||||
the best prefill speed with no cost to decode.
|
||||
|
||||
If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
|
||||
(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
|
||||
If stability headroom matters most, use `ncmoe 34 - ubatch 1536` (about 2.0
|
||||
GiB headroom).
|
||||
|
||||
## Limitations
|
||||
|
||||
|
||||
Reference in new issue
Block a user