dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-primary: util 0.90 — production bf16 boot OOM fix

Browse Source 
The 0.98 boot died at KV allocation: torch OOM tried 10.28 GiB (the exact
3x262144 bf16 pool = 11,047,010,304 bytes) with 9.46 GiB free per rank.
The rootless bench probe had booted the same numbers; the root unit's
pinned-expert path holds ~0.8 GiB/rank more. Util 0.90 returns ~2.5 GiB/rank
of workspace with the pool ask unchanged; measured KV capacity law
(798,280 tokens at 11.05e9) unchanged.
This commit is contained in:
Frank
2026-10-01 02:12:54 -07:00
parent c1248d4a5d
commit e8524d4046
2 changed files with 14 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-primary.env
+1 -1
View File
@@ -67,4 +67,4 @@ FRANK_PRIMARY_KV_DTYPE=auto
FRANK_PRIMARY_MNBT=1024
FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64
FRANK_PRIMARY_UTIL=0.98
FRANK_PRIMARY_UTIL=0.90
tests/frank_primary_unit.bats
+13 -8
View File
@@ -74,16 +74,21 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
}
@test "the qualified engine shape is in the env file" {
# 2026-10-01 boot probe: 262,144 x 3 seqs at bf16 KV, util 0.98,
# KV_BYTES 11,047,010,304 PER RANK boots at "KV cache size: 798,280
# tokens" and reports max concurrency 3.05x for 262,144-token requests
# — all three full windows fit with an 11,848-token margin. The old
# "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 drafter's
# workspace reservation, not a VRAM wall: no-spec takes 11.05 GiB/rank
# and boots clean.
# 2026-10-01 boot probe: 262,144 x 3 seqs at bf16 KV, KV_BYTES
# 11,047,010,304 PER RANK boots at "KV cache size: 798,280 tokens" and
# reports max concurrency 3.05x for 262,144-token requests — all three
# full windows fit with an 11,848-token margin. The old "KV_BYTES is
# PER RANK, 9/16 GiB OOM" law was the MTP4 drafter's workspace
# reservation, not a VRAM wall: no-spec takes 11.05 GiB/rank and boots
# clean.
# UTIL 0.90 is a PRODUCTION-corrected value: the first production boot
# at 0.98 died at KV allocation (torch OOM, tried 10.28 GiB with 9.46
# free — 0.82 GiB short) while the rootless bench probe at the same
# numbers had booted; util 0.90 returns ~2.4 GiB/rank of workspace and
# is the deployed value.
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.90$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=11047010304$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"