From de4d22785d4378e119a9691d7d9fcd0a17f3b267 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 26 Sep 2026 21:01:12 -0700 Subject: [PATCH] frank-primary runbook: record the KV law and per-ctx VRAM costs 29,696 B/token ctx-scaled (12 of 48 layers carry KV - full_attention_interval=4; indexer and draft KV included); weights/RS/compute are ctx-independent. 262k = 18.75 GiB total / 9.37 per card; 100k = 14.3 / 7.1. Parallel slots split the pool rather than multiplying it. --- .../LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md index ffc112e..f4d50aa 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md +++ b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md @@ -142,6 +142,14 @@ answers; after any aborted deploy, confirm the responder is the unit's own MainPID), and llama-server's SIGTERM cleanup hung after spec-decode service (logged `cleaning up before exit` and never exited — `kill -9` was needed; `systemctl stop` relies on TimeoutStopSec)._ +**KV law (from the -lv 4 load log, cross-checked by the GGUF header):** +29,696 B/token total ctx-scaled VRAM — 24,576 B main KV (only 12 of 48 +layers carry attention, `full_attention_interval=4`; 2 KV heads x 256, +f16) + 3,072 B indexer + 2,048 B draft KV. Everything else (weights +~6.7 GiB, recurrent state 0.33 GiB, compute buffers ~4.1 GiB) is +ctx-independent. At 262,144: 7.25 GiB scaled, 18.75 GiB total, 9.37 +GiB/card. At 100,000: 2.77 GiB scaled, 14.3 GiB total, 7.1 GiB/card. +`--parallel N` splits the same pool — slots do not multiply KV. ## Rollback