docs: record dual-f16 KV refit on frank-default-small (2026-08-14)

This commit is contained in:
Frank committed 2026-08-14 12:59:10 -07:00
1 parent f96aa2b692
commit d72092b2c7
4 files changed
+26 -11

No files matched your search

+8
View File
@@ -102,6 +102,14 @@ It samples GPU VRAM and process RAM every 10 seconds and appends a CSV
row. The columns are timestamp, uptime, GPU VRAM MiB, GPU utilization,
process RSS, process VSZ, and systemd MemoryPeak.
## Resolution: dual-f16 KV Cache (2026-08-14)
The fix chose the first mechanism over the second. Both K and V moved to f16 on 2026-08-14. When K is already f16, the flash-attention MMA_F16 prefill kernel needs no F16 conversion copy, so the context-proportional workspace disappears. The pool becomes static: 20,480 bytes per cell (10 full-attention layers x 2 kv heads x 256 dims x 2 bytes for K plus 2 bytes for V).
The pool at 800768 cells cost 15,640 MiB and did not fit. The load-time warmup graph (1472x1472 image) failed cudaMalloc by 248 MiB, and the service idled at 23,966 of 24,576 MiB. The context pool was refitted to 700416 cells (4 slots x 175104): KV pool 13,680 MiB, idle 22,110 MiB, about 2.0 GiB free after warmup. Parallel stays 4.
Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode about 42-44 t/s, cached prompt tokens working. The vram monitor log shows the restart arc (23,966 -> 5,210 during load -> 22,110 idle).
## Mitigation Options
The documentation for the benchmark ([[frank-default-small-benchmark]])