docs: record dual-f16 KV refit on frank-default-small (2026-08-14)
This commit is contained in:
1 parent
f96aa2b692
commit
d72092b2c7
4 files changed
+26
-11
No files matched your search
@@ -102,6 +102,14 @@ It samples GPU VRAM and process RAM every 10 seconds and appends a CSV
|
||||
row. The columns are timestamp, uptime, GPU VRAM MiB, GPU utilization,
|
||||
process RSS, process VSZ, and systemd MemoryPeak.
|
||||
|
||||
## Resolution: dual-f16 KV Cache (2026-08-14)
|
||||
|
||||
The fix chose the first mechanism over the second. Both K and V moved to f16 on 2026-08-14. When K is already f16, the flash-attention MMA_F16 prefill kernel needs no F16 conversion copy, so the context-proportional workspace disappears. The pool becomes static: 20,480 bytes per cell (10 full-attention layers x 2 kv heads x 256 dims x 2 bytes for K plus 2 bytes for V).
|
||||
|
||||
The pool at 800768 cells cost 15,640 MiB and did not fit. The load-time warmup graph (1472x1472 image) failed cudaMalloc by 248 MiB, and the service idled at 23,966 of 24,576 MiB. The context pool was refitted to 700416 cells (4 slots x 175104): KV pool 13,680 MiB, idle 22,110 MiB, about 2.0 GiB free after warmup. Parallel stays 4.
|
||||
|
||||
Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode about 42-44 t/s, cached prompt tokens working. The vram monitor log shows the restart arc (23,966 -> 5,210 during load -> 22,110 idle).
|
||||
|
||||
## Mitigation Options
|
||||
|
||||
The documentation for the benchmark ([[frank-default-small-benchmark]])
|
||||
|
||||
Reference in new issue
Block a user