frank-primary: 3 full windows, pool at the contract floor, MNBT 2048
Frank's call 2026-10-01: MAXLEN 262144 x SEQS 3 = 786,432 tokens of contract. Measured fp8 KV costs 7,698 B/token (4-slot boot 8,311,013,376 B -> 1,079,582 tokens; 5-slot boot 10,388,766,720 B -> 1,350,182 tokens), so the pool floor for the contract is 6.054e9 B. Pinned at 6,100,000,000 (~792k tokens, ~5.9k margin) so the remaining ~2.2 GiB versus the 4-slot shape goes to the expert ARENA - the fork sizes its decode-critical expert cache from the remainder after weights, pool and workspace. MNBT 2048 (Frank): the arena sweep measured 6,499 slots at every rung <= 4096, so the larger batch costs the arena nothing. Seam retargeted (3 seqs / 6.1e9 / MNBT 2048); docs + AGENTS row + log updated, including the closed DCP investigation.
This commit is contained in:
1 parent
6bbcaf3999
commit
a1b650aabc
5 files changed
+51
-26
No files matched your search
@@ -278,8 +278,9 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=262144 x SEQS 4 (1,048,576 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=8311013376 PER RANK (fp8, pool scales with windows)
|
||||
FRANK_PRIMARY_MAXLEN=262144 x SEQS 3 (786,432 tokens of contract)
|
||||
FRANK_PRIMARY_KV_BYTES=6100000000 PER RANK (fp8, pinned to the CONTRACT
|
||||
FLOOR so the remainder feeds the arena)
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (K and V together)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
@@ -298,7 +299,8 @@ memory environment). The night searched two concurrency contracts:
|
||||
| **3 × 241664, bf16** | **PASS — KV 732,359 tokens, 3.03x** (max possible at 3 contexts) |
|
||||
| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
|
||||
| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
|
||||
| **4 × 262144, fp8** | **DEPLOYED.** Five slots booted but left 0 bytes free at runtime: a real client request (995-token prompt, `max_tokens=261149`) killed the worker (`Tried to allocate 50.00 MiB ... 0 bytes free`). Dropping one slot returns ~1.93 GiB per rank for activations and graph capture |
|
||||
| 4 × 262144, fp8 | runtime-survivable stopgap — KV 1,079,582 tokens / 4.12x |
|
||||
| **3 × 262144, fp8, pool at the floor** | **DEPLOYED** (Frank's call): 786,432 tokens of contract; pool pinned at 6,100,000,000 B/rank (the measured 7,698 B/token floor is 6.054e9) so the ~2.2 GiB difference versus the 4-slot shape goes to the expert ARENA instead of unused KV |
|
||||
|
||||
Boot is NOT proof of runtime survivability: the five-slot shape passed every
|
||||
boot gate and still died on the first long-generation request. After any
|
||||
|
||||
@@ -737,6 +737,17 @@
|
||||
"Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
|
||||
down until a human noticed. An explicit `systemctl stop` still wins, so
|
||||
bench windows and deploys are unaffected.
|
||||
- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR,
|
||||
MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of
|
||||
contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor
|
||||
and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing
|
||||
the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the
|
||||
decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the
|
||||
arena nothing (the sweep measured 6,499 slots at every rung <= 4096).
|
||||
- **DCP investigated and closed** (see the parallelism survey in the runbook):
|
||||
`--decode-context-parallel-size 2` is rejected at boot — DCP for GQA requires
|
||||
TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning
|
||||
knob on this box.
|
||||
- Production bring-up: the shape boots and serves on :7770 — engine log
|
||||
verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
|
||||
contract probes (reasoning split, tool_calls). Rollback target for the
|
||||
|
||||
Reference in new issue
Block a user