frank-primary: 2 bf16 windows + --expert-precise-memory

Both of Frank's changes at once.

KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144).
bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280
tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's
6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9
(~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank.

--expert-precise-memory selects the fitted overhead (5.25 GiB) over the default
(7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's
 term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the
cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so
MNBT 2048 pays nothing.

Seam + docs + AGENTS row + log updated.
This commit is contained in:
Frank committed 2026-10-01 13:52:13 -07:00
1 parent 3f1860aa5f
commit 389b605baf
6 files changed
+82 -41

No files matched your search

@@ -278,10 +278,12 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
```
FRANK_PRIMARY_MAXLEN=262144 x SEQS 3 (786,432 tokens of contract)
FRANK_PRIMARY_KV_BYTES=6100000000 PER RANK (fp8, pinned to the CONTRACT
FLOOR so the remainder feeds the arena)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (K and V together)
FRANK_PRIMARY_MAXLEN=262144 x SEQS 2 (524,288 tokens of contract)
FRANK_PRIMARY_KV_BYTES=7300000000 PER RANK (bf16 = auto; 13,839 B/token)
FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized)
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (fitted overhead
5.25 GiB instead of the 7.60 GiB default,
so ~1,025 more arena slots)
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
@@ -300,7 +302,8 @@ memory environment). The night searched two concurrency contracts:
| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
| 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x |
| **3 × 262144, fp8, pool at the floor** | **DEPLOYED** (Frank's call): 786,432 tokens of contract; pool pinned at 6,100,000,000 B/rank (the measured 7,698 B/token floor is 6.054e9) so the ~2.2 GiB difference versus the 4-slot shape goes to the expert ARENA instead of unused KV |
| 3 × 262144, fp8, pool at the floor | KV 792,069 tokens / 3.02x; arena 6,248 slots; 3.59 GiB free VRAM measured |
| **2 × 262144, bf16 + `--expert-precise-memory`** | **DEPLOYED** (Frank's call): unquantized KV at the full native window, two contexts; pool 7.3e9 B/rank (bf16 rate 13,839 B/token measured: 12,276 attention + 1,563 mamba/side, vs fp8's 6,138 + 1,563 = 7,701); the precise-memory flag drops the plan's overhead 7.60 -> 5.25 GiB so the arena grows from 6,248 to ~7,278 slots |
Boot is NOT proof of runtime survivability: the five-slot shape passed every
boot gate and still died on the first long-generation request. After any