docs: record dual-f16 KV refit on frank-default-small (2026-08-14)

This commit is contained in:
Frank committed 2026-08-14 12:59:10 -07:00
1 parent f96aa2b692
commit d72092b2c7
4 files changed
+26 -11

No files matched your search

+15 -8
View File
@@ -76,17 +76,21 @@ Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
## VRAM Budget Model
The live service holds a fixed 800000-token KV pool plus the mmproj vision
tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a
24 GiB budget:
The live service holds a fixed 700416-token KV pool (f16/f16 since
2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA
overhead. The model below fits llama-bench peak VRAM into a 24 GiB
budget:
```
service VRAM = llama-bench peak + KV pool (11.44 GiB) + overhead (2.2 GiB)
service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB)
```
This model reproduces the historical OOMs. Configurations whose sum exceeds
24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about 24.4 GiB
and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
This model reproduces the historical OOMs. Configurations whose sum
exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about
24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and
`33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at
800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the
service runs 700416 cells (13,680 MiB).
## Candidate Configurations
@@ -111,7 +115,10 @@ moves one expert layer to CPU for stability:
```
This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
44.4 t/s, and about 3.0 GiB of headroom at the 800000-token pool.
44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool
(2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and
removes the context-proportional flash-attn F16-conversion workspace that
caused the cuBLAS crash class).
`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory