frank-primary runbook: record the KV law and per-ctx VRAM costs
29,696 B/token ctx-scaled (12 of 48 layers carry KV - full_attention_interval=4; indexer and draft KV included); weights/RS/compute are ctx-independent. 262k = 18.75 GiB total / 9.37 per card; 100k = 14.3 / 7.1. Parallel slots split the pool rather than multiplying it.
This commit is contained in:
1 parent
82ff9dda20
commit
de4d22785d
1 file changed
+8
@@ -142,6 +142,14 @@ answers; after any aborted deploy, confirm the responder is the unit's
|
|||||||
own MainPID), and llama-server's SIGTERM cleanup hung after
|
own MainPID), and llama-server's SIGTERM cleanup hung after
|
||||||
spec-decode service (logged `cleaning up before exit` and never exited —
|
spec-decode service (logged `cleaning up before exit` and never exited —
|
||||||
`kill -9` was needed; `systemctl stop` relies on TimeoutStopSec)._
|
`kill -9` was needed; `systemctl stop` relies on TimeoutStopSec)._
|
||||||
|
**KV law (from the -lv 4 load log, cross-checked by the GGUF header):**
|
||||||
|
29,696 B/token total ctx-scaled VRAM — 24,576 B main KV (only 12 of 48
|
||||||
|
layers carry attention, `full_attention_interval=4`; 2 KV heads x 256,
|
||||||
|
f16) + 3,072 B indexer + 2,048 B draft KV. Everything else (weights
|
||||||
|
~6.7 GiB, recurrent state 0.33 GiB, compute buffers ~4.1 GiB) is
|
||||||
|
ctx-independent. At 262,144: 7.25 GiB scaled, 18.75 GiB total, 9.37
|
||||||
|
GiB/card. At 100,000: 2.77 GiB scaled, 14.3 GiB total, 7.1 GiB/card.
|
||||||
|
`--parallel N` splits the same pool — slots do not multiply KV.
|
||||||
|
|
||||||
## Rollback
|
## Rollback
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user