frank-primary: 5 x full 262144 contexts at fp8 (final shape)
Frank's call after the night ladder: five contexts at the model's full native 262,144 window beat six shortened ones. Six full-length contexts do NOT fit at any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the 6-seq graph/activation footprint leaves too little for the pool); 5 x 262144 at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured 'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens per request: 5.15x' (39,462 tokens of margin). Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144) re-pinned.
This commit is contained in:
1 parent
7cab15ac39
commit
bd311bf591
6 files changed
+83
-61
No files matched your search
@@ -75,20 +75,20 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
}
|
||||
|
||||
@test "the qualified engine shape is in the env file" {
|
||||
# 2026-10-01 night ladder (Frank's method: subtract until it boots, then
|
||||
# one bonus step): 262144 OOMed at graph capture (76 MiB short, 0 bytes
|
||||
# free), 245760 OOMed identically, 237568 PASSED (KV 718,606 tokens /
|
||||
# 3.0x), and the +4096 bonus attempt 241664 PASSED and IS production
|
||||
# (KV 732,359 tokens / 3.03x at 241,664-token requests, 7,367-token
|
||||
# margin over the 3-window contract). UTIL stays 0.98 — the production
|
||||
# unit's pinned-expert path needs the workspace the bench probe hid;
|
||||
# 0.98 with the smaller window fits. KV_BYTES 10,184,000,000 scales the
|
||||
# pool with the window.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=241664$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
# 2026-10-01 night ladder (Frank's method). Two contracts were searched:
|
||||
# SEQS=3 bf16 topped out at 241664 (262144/245760 OOMed at graph capture;
|
||||
# KV 732,359 tokens / 3.03x). SEQS=6 fp8 did NOT fit — 6 x 241664 tried
|
||||
# 10.70 GiB against 9.18 GiB free, so six full-length contexts are
|
||||
# impossible here. SEQS=5 fp8 at the model's FULL native 262144 window
|
||||
# PASSED on the first rung and is DEPLOYED: KV 1,350,182 tokens,
|
||||
# Maximum concurrency 5.15x, 39,462 tokens of margin over the
|
||||
# 1,310,720-token contract. UTIL stays 0.98 (lowering it shrinks the
|
||||
# engine's own budget instead of buying slack).
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=5$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=10184000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=10388766720$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
@@ -151,13 +151,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
! grep -q -- '--speculative-config' "$SERVE"
|
||||
}
|
||||
|
||||
@test "bf16 KV is PRESENT — clean cache won the trade" {
|
||||
# --kv-cache-dtype auto = bf16 attention KV. Doubles attention-KV
|
||||
# bytes/token (measured 14,047 vs 7,926) and costs ~14% decode vs fp8
|
||||
# at 4064 ctx (49.25 -> 36.42 with the gather win) — traded deliberately
|
||||
# for clean cache and 262k windows (Frank's call). The fp8 shape is
|
||||
# restorable from git tag frank-primary-fp8-mtp-20261001.
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
|
||||
@test "fp8 KV is PRESENT — capacity won over the bf16 decode edge" {
|
||||
# --kv-cache-dtype fp8 (the stock flag quantizes K and V together).
|
||||
# Capacity is what buys the five full 262,144-token contexts: fp8 costs
|
||||
# 7,926 B/token against bf16's ~13,900-14,047, so 5x262144 fits in
|
||||
# 10,388,766,720 B/rank (KV 1,350,182 tokens / 5.15x) where bf16 topped
|
||||
# out at three 241,664-token contexts. The bf16 ~4% decode edge was
|
||||
# traded away for it (Frank's call, 2026-10-01).
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user