frank-primary runbook: record the KV law and per-ctx VRAM costs
29,696 B/token ctx-scaled (12 of 48 layers carry KV - full_attention_interval=4; indexer and draft KV included); weights/RS/compute are ctx-independent. 262k = 18.75 GiB total / 9.37 per card; 100k = 14.3 / 7.1. Parallel slots split the pool rather than multiplying it.
This commit is contained in:
@@ -142,6 +142,14 @@ answers; after any aborted deploy, confirm the responder is the unit's
|
||||
own MainPID), and llama-server's SIGTERM cleanup hung after
|
||||
spec-decode service (logged `cleaning up before exit` and never exited —
|
||||
`kill -9` was needed; `systemctl stop` relies on TimeoutStopSec)._
|
||||
**KV law (from the -lv 4 load log, cross-checked by the GGUF header):**
|
||||
29,696 B/token total ctx-scaled VRAM — 24,576 B main KV (only 12 of 48
|
||||
layers carry attention, `full_attention_interval=4`; 2 KV heads x 256,
|
||||
f16) + 3,072 B indexer + 2,048 B draft KV. Everything else (weights
|
||||
~6.7 GiB, recurrent state 0.33 GiB, compute buffers ~4.1 GiB) is
|
||||
ctx-independent. At 262,144: 7.25 GiB scaled, 18.75 GiB total, 9.37
|
||||
GiB/card. At 100,000: 2.77 GiB scaled, 14.3 GiB total, 7.1 GiB/card.
|
||||
`--parallel N` splits the same pool — slots do not multiply KV.
|
||||
|
||||
## Rollback
|
||||
|
||||
|
||||
Reference in New Issue
Block a user