docs: record dual-f16 KV refit on frank-default-small (2026-08-14)
This commit is contained in:
1 parent
f96aa2b692
commit
d72092b2c7
4 files changed
+26
-11
No files matched your search
@@ -76,17 +76,21 @@ Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
|
||||
|
||||
## VRAM Budget Model
|
||||
|
||||
The live service holds a fixed 800000-token KV pool plus the mmproj vision
|
||||
tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a
|
||||
24 GiB budget:
|
||||
The live service holds a fixed 700416-token KV pool (f16/f16 since
|
||||
2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA
|
||||
overhead. The model below fits llama-bench peak VRAM into a 24 GiB
|
||||
budget:
|
||||
|
||||
```
|
||||
service VRAM = llama-bench peak + KV pool (11.44 GiB) + overhead (2.2 GiB)
|
||||
service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB)
|
||||
```
|
||||
|
||||
This model reproduces the historical OOMs. Configurations whose sum exceeds
|
||||
24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about 24.4 GiB
|
||||
and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
|
||||
This model reproduces the historical OOMs. Configurations whose sum
|
||||
exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about
|
||||
24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and
|
||||
`33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at
|
||||
800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the
|
||||
service runs 700416 cells (13,680 MiB).
|
||||
|
||||
## Candidate Configurations
|
||||
|
||||
@@ -111,7 +115,10 @@ moves one expert layer to CPU for stability:
|
||||
```
|
||||
|
||||
This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
|
||||
44.4 t/s, and about 3.0 GiB of headroom at the 800000-token pool.
|
||||
44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool
|
||||
(2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and
|
||||
removes the context-proportional flash-attn F16-conversion workspace that
|
||||
caused the cuBLAS crash class).
|
||||
|
||||
`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
|
||||
`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory
|
||||
|
||||
Reference in new issue
Block a user