frank-primary: 2 bf16 windows + --expert-precise-memory
Both of Frank's changes at once. KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144). bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280 tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's 6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9 (~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank. --expert-precise-memory selects the fitted overhead (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so MNBT 2048 pays nothing. Seam + docs + AGENTS row + log updated.
This commit is contained in:
1 parent
3f1860aa5f
commit
389b605baf
6 files changed
+82
-41
No files matched your search
@@ -115,6 +115,7 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
||||
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
||||
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
|
||||
${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
|
||||
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
|
||||
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
||||
|
||||
Reference in new issue
Block a user