frank-primary: 2 bf16 windows + --expert-precise-memory
Both of Frank's changes at once. KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144). bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280 tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's 6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9 (~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank. --expert-precise-memory selects the fitted overhead (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so MNBT 2048 pays nothing. Seam + docs + AGENTS row + log updated.
This commit is contained in:
1 parent
3f1860aa5f
commit
389b605baf
6 files changed
+82
-41
No files matched your search
@@ -278,10 +278,12 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=262144 x SEQS 3 (786,432 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=6100000000 PER RANK (fp8, pinned to the CONTRACT
|
||||
FLOOR so the remainder feeds the arena)
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (K and V together)
|
||||
FRANK_PRIMARY_MAXLEN=262144 x SEQS 2 (524,288 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=7300000000 PER RANK (bf16 = auto; 13,839 B/token)
|
||||
FRANK_PRIMARY_KV_DTYPE=auto --kv-cache-dtype auto (bf16, unquantized)
|
||||
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (fitted overhead
|
||||
5.25 GiB instead of the 7.60 GiB default,
|
||||
so ~1,025 more arena slots)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
NO --speculative-config, NO --expert-prefetch
|
||||
@@ -300,7 +302,8 @@ memory environment). The night searched two concurrency contracts:
|
||||
| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
|
||||
| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
|
||||
| 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x |
|
||||
| **3 × 262144, fp8, pool at the floor** | **DEPLOYED** (Frank's call): 786,432 tokens of contract; pool pinned at 6,100,000,000 B/rank (the measured 7,698 B/token floor is 6.054e9) so the ~2.2 GiB difference versus the 4-slot shape goes to the expert ARENA instead of unused KV |
|
||||
| 3 × 262144, fp8, pool at the floor | KV 792,069 tokens / 3.02x; arena 6,248 slots; 3.59 GiB free VRAM measured |
|
||||
| **2 × 262144, bf16 + `--expert-precise-memory`** | **DEPLOYED** (Frank's call): unquantized KV at the full native window, two contexts; pool 7.3e9 B/rank (bf16 rate 13,839 B/token measured: 12,276 attention + 1,563 mamba/side, vs fp8's 6,138 + 1,563 = 7,701); the precise-memory flag drops the plan's overhead 7.60 -> 5.25 GiB so the arena grows from 6,248 to ~7,278 slots |
|
||||
|
||||
Boot is NOT proof of runtime survivability: the five-slot shape passed every
|
||||
boot gate and still died on the first long-generation request. After any
|
||||
|
||||
Reference in new issue
Block a user