frank-primary: 2 bf16 windows + --expert-precise-memory

Both of Frank's changes at once.

KV is bf16 (--kv-cache-dtype auto) at two full-size windows (2 x 262144).
bf16 measures 13,839 B/token (bf16 3x241664 boot: 11,047,010,304 B -> 798,280
tokens), decomposing as 12,276 attention + 1,563 mamba/side caches vs fp8's
6,138 + 1,563 = 7,701. 524,288 tokens needs 7.256e9 B; pinned at 7.3e9
(~527,500 tokens, 3,206 margin). Pool cost: 5.68 -> 6.76 GiB per rank.

--expert-precise-memory selects the fitted overhead (5.25 GiB) over the default
(7.60 GiB = fitted + compile/graph-capture headroom). It moves only the plan's
 term, so the expert arena goes 6,248 -> ~7,278 slots (~1,025 more) at the
cost of boot-time capture slack. Overhead adds 0.22 GiB/1024 nbt above 4096, so
MNBT 2048 pays nothing.

Seam + docs + AGENTS row + log updated.
This commit is contained in:
Frank committed 2026-10-01 13:52:13 -07:00
1 parent 3f1860aa5f
commit 389b605baf
6 files changed
+82 -41

No files matched your search

+24 -15
View File
@@ -100,16 +100,18 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# deployed shape: ~1.93 GiB more headroom per rank for
# activations/graphs. UTIL stays 0.98 (lowering it shrinks the engine's
# own budget instead of buying slack).
# 3 x 262144 = 786,432 tokens; at the measured 7,698 B/token the pool
# floor is 6.054e9 B, pinned at 6.1e9 so the margin is ~5.9k tokens and
# EVERYTHING ELSE goes to the expert arena (the decode-critical expert
# cache is the remainder after weights, pool and workspace).
# FINAL (2026-10-01, Frank): TWO full-size windows with BF16 KV plus
# --expert-precise-memory. bf16 costs 13,839 B/token (measured), so
# 2 x 262144 = 524,288 tokens needs 7.256e9 B; pinned at 7.3e9 (~527.5k
# tokens, 3,206 margin). The precise flag drops the plan's overhead
# 7.60 -> 5.25 GiB, moving the arena from 6,248 to ~7,278 slots.
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=6100000000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=7300000000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
@@ -132,6 +134,12 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).
grep -q -- '--decode-context-parallel-size "$FRANK_PRIMARY_DCP"' "$SERVE"
! grep -q '^FRANK_PRIMARY_DCP=' "$ENV_FILE"
# --expert-precise-memory selects the fitted overhead (5.25 GiB) over the
# default (7.60 GiB = fitted + compile/graph-capture headroom), which only
# moves the plan's `room` term and therefore the expert arena: ~1,025
# more slots at 2.344 MiB each.
grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
}
@test "tools and reasoning are wired — the client contract is served" {
@@ -175,14 +183,15 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
! grep -q -- '--speculative-config' "$SERVE"
}
@test "fp8 KV is PRESENT — capacity won over the bf16 decode edge" {
# --kv-cache-dtype fp8 (the stock flag quantizes K and V together).
# Capacity is what buys the five full 262,144-token contexts: fp8 costs
# 7,926 B/token against bf16's ~13,900-14,047, so 5x262144 fits in
# 10,388,766,720 B/rank (KV 1,350,182 tokens / 5.15x) where bf16 topped
# out at three 241,664-token contexts. The bf16 ~4% decode edge was
# traded away for it (Frank's call, 2026-10-01).
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
@test "KV dtype is bf16 (auto) — exactness, paid for in pool bytes" {
# --kv-cache-dtype auto = bf16 (unquantized). Frank's call 2026-10-01.
# bf16 costs 13,839 B/token against fp8's 7,701 — decomposed as 12,276
# attention + 1,563 mamba/side (dtype-independent) versus fp8's 6,138 +
# 1,563. That halves the pool capacity per byte, which is why the contract
# is two full-size windows instead of three; in exchange the KV is exact
# (no quantization error) and measured +4% decode at shallow depth, where
# fp8's dequant cost dominates over the extra bytes read.
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
}