frank-primary: 2 bf16 windows + --expert-precise-memory

Both of Frank's changes at once.

KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144).
bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280
tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's
6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9
(~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank.

--expert-precise-memory selects the fitted overhead (5.25 GiB) over the default
(7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's
 term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the
cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so
MNBT 2048 pays nothing.

Seam + docs + AGENTS row + log updated.
This commit is contained in:
Frank committed 2026-10-01 13:52:13 -07:00
1 parent 3f1860aa5f
commit 389b605baf
6 files changed
+82 -41

No files matched your search

+1
View File
@@ -115,6 +115,7 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \