frank-primary: 2 bf16 windows + --expert-precise-memory

Both of Frank's changes at once.

KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144).
bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280
tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's
6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9
(~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank.

--expert-precise-memory selects the fitted overhead (5.25 GiB) over the default
(7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's
 term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the
cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so
MNBT 2048 pays nothing.

Seam + docs + AGENTS row + log updated.
This commit is contained in:
Frank committed 2026-10-01 13:52:13 -07:00
1 parent 3f1860aa5f
commit 389b605baf
6 files changed
+82 -41

No files matched your search

+23 -20
View File
@@ -59,28 +59,31 @@ FRANK_PRIMARY_PUBLISH=7770:7770
# hoc experiment identity (both hold the GPUs / name registrations).
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
# --- qualified engine shape (2026-10-01, final: 3 windows + max arena) --------
# SEQS 3 x MAXLEN 262144 at fp8 KV (Frank's call, 2026-10-01): three contexts
# of the model's FULL native window, with the KV pool pinned to the FLOOR the
# contract needs so that everything else goes to the expert ARENA — the
# remainder is what the fork's planner gives the decode-critical expert cache.
# 3 x 262144 = 786,432 tokens; measured fp8 rate 7,698 B/token (from the
# 4-slot boot: 8,311,013,376 B -> 1,079,582 tokens, and the 5-slot boot:
# 10,388,766,720 B -> 1,350,182 tokens) => 6.054e9 B is the floor. Pinned at
# 6,100,000,000 => ~792k tokens, a ~5.9k-token margin, and ~2.2 GiB returned
# to the arena versus the 4-slot shape.
# History in this shape: 5 x 262144 booted (KV 1,350,182 / 5.15x) but left 0
# bytes free and a real request (995-token prompt, max_tokens=261149) killed
# the worker; 4 x 262144 (KV 1,079,582 / 4.12x) was the runtime-survivable
# stopgap. Six full-length contexts never fit (6 x 241664 asked 10.70 GiB
# against 9.18 GiB free).
# MNBT 2048 (Frank's call): the arena sweep measured it at the same 6,499-slot
# ceiling as <=4096, so the bigger batch costs the arena nothing.
# --- qualified engine shape (2026-10-01, final: 2 bf16 windows + precise mem) --
# SEQS 2 x MAXLEN 262144 at BF16 KV (Frank's call, 2026-10-01): two contexts of
# the model's FULL native window with UNQUANTIZED KV, plus
# --expert-precise-memory to hand the arena the compile/graph-capture headroom.
# Pool: bf16 costs 13,839 B/token (measured: 11,047,010,304 B -> 798,280 tokens
# on the 3x241664 bf16 boot; decomposes as 12,276 attention + 1,563 mamba/side
# caches, against fp8's 6,138 + 1,563 = 7,701). 2 x 262144 = 524,288 tokens
# -> 7.256e9 B floor, pinned at 7,300,000,000 (~527,500 tokens, 3,206 margin).
# Budget (per rank): 31.22 GiB total; non-expert weights 6.28, graphs 0.55,
# non-torch ~1.45, pool 6.76 -> ~29.35 used, ~2.5 GiB free (measured basis:
# the fp8 3-window shape used 28.27 and left 3.59).
# PRECISE MEMORY: the registry's default overhead for this architecture is
# 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2 nbt4096/seqs16/len262144, heavier
# rank 5.41) PLUS 2.35 GiB of compile/graph-capture headroom. The flag selects
# the fitted figure, which only moves the plan's `room` and therefore the arena:
# fixed 16.91 -> 14.56 GiB, arena 6,248 -> ~7,278 slots (~2.35 GiB = ~1,025
# slots at 2.344 MiB/slot). The risk it buys back is boot-time capture slack.
# MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so
# 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096.
FRANK_PRIMARY_MAXLEN=262144
FRANK_PRIMARY_SEQS=3
FRANK_PRIMARY_KV_BYTES=6100000000
FRANK_PRIMARY_KV_DTYPE=fp8
FRANK_PRIMARY_SEQS=2
FRANK_PRIMARY_KV_BYTES=7300000000
FRANK_PRIMARY_KV_DTYPE=auto
FRANK_PRIMARY_MNBT=2048
FRANK_PRIMARY_PRECISE_MEMORY=1
FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64
FRANK_PRIMARY_UTIL=0.98