frank-primary: 5 x full 262144 contexts at fp8 (final shape)

Frank's call after the night ladder: five contexts at the model's full native
262,144 window beat six shortened ones. Six full-length contexts do NOT fit at
any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the
6-seq graph/activation footprint leaves too little for the pool); 5 x 262144
at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured
'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens
per request: 5.15x' (39,462 tokens of margin).

Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured
earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling
law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144)
re-pinned.
This commit is contained in:
Frank committed 2026-10-01 03:52:10 -07:00
1 parent 7cab15ac39
commit bd311bf591
6 files changed
+83 -61

No files matched your search

+21 -20
View File
@@ -75,20 +75,20 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
}
@test "the qualified engine shape is in the env file" {
# 2026-10-01 night ladder (Frank's method: subtract until it boots, then
# one bonus step): 262144 OOMed at graph capture (76 MiB short, 0 bytes
# free), 245760 OOMed identically, 237568 PASSED (KV 718,606 tokens /
# 3.0x), and the +4096 bonus attempt 241664 PASSED and IS production
# (KV 732,359 tokens / 3.03x at 241,664-token requests, 7,367-token
# margin over the 3-window contract). UTIL stays 0.98 — the production
# unit's pinned-expert path needs the workspace the bench probe hid;
# 0.98 with the smaller window fits. KV_BYTES 10,184,000,000 scales the
# pool with the window.
grep -q '^FRANK_PRIMARY_MAXLEN=241664$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
# 2026-10-01 night ladder (Frank's method). Two contracts were searched:
# SEQS=3 bf16 topped out at 241664 (262144/245760 OOMed at graph capture;
# KV 732,359 tokens / 3.03x). SEQS=6 fp8 did NOT fit — 6 x 241664 tried
# 10.70 GiB against 9.18 GiB free, so six full-length contexts are
# impossible here. SEQS=5 fp8 at the model's FULL native 262144 window
# PASSED on the first rung and is DEPLOYED: KV 1,350,182 tokens,
# Maximum concurrency 5.15x, 39,462 tokens of margin over the
# 1,310,720-token contract. UTIL stays 0.98 (lowering it shrinks the
# engine's own budget instead of buying slack).
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=5$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=10184000000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=10388766720$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
@@ -151,13 +151,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
! grep -q -- '--speculative-config' "$SERVE"
}
@test "bf16 KV is PRESENT — clean cache won the trade" {
# --kv-cache-dtype auto = bf16 attention KV. Doubles attention-KV
# bytes/token (measured 14,047 vs 7,926) and costs ~14% decode vs fp8
# at 4064 ctx (49.25 -> 36.42 with the gather win) — traded deliberately
# for clean cache and 262k windows (Frank's call). The fp8 shape is
# restorable from git tag frank-primary-fp8-mtp-20261001.
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
@test "fp8 KV is PRESENT — capacity won over the bf16 decode edge" {
# --kv-cache-dtype fp8 (the stock flag quantizes K and V together).
# Capacity is what buys the five full 262,144-token contexts: fp8 costs
# 7,926 B/token against bf16's ~13,900-14,047, so 5x262144 fits in
# 10,388,766,720 B/rank (KV 1,350,182 tokens / 5.15x) where bf16 topped
# out at three 241,664-token contexts. The bf16 ~4% decode edge was
# traded away for it (Frank's call, 2026-10-01).
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
}