frank-primary: record the measured shape-D numbers (398,658 tokens, 1.99x, 6,496 slots)

Comments/docs only - no functional change. The live boot of c769ff2 reported:
  GPU KV cache size: 398,658 tokens,
  Maximum concurrency for 200,704 tokens per request: 1.99x
  expert cache space 14.87 GiB / 6,496 slots of 2.344 MiB

The arena prediction (6,496) landed exactly; the concurrency prediction (2.04x)
did not, because the per-token KV price is SHAPE-DEPENDENT: 7,701 B/token at
MAXLEN 262144 (6.1e9 -> 792,069 tokens) but 7,892 at 200704 (3.146e9 ->
398,658). Never extrapolate a concurrency figure across geometries.

Also smoke-verified: three concurrent 67-token requests answered in parallel,
~4.1 s each - the 3 slots are real, and the pool admits 2 full-length.
This commit is contained in:
Frank committed 2026-10-02 11:48:09 -07:00
1 parent c769ff2f63
commit 295d5784d3
5 files changed
+39 -28

No files matched your search

+7 -5
View File
@@ -105,9 +105,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# max_num_reqs + 1, so the width is SEQS + 1 and MNBT only moves the
# compile RANGE.
# So the shape is 3 slots with a pool that admits only ~2 full-length
# conversations: fp8 measures 7,701 B/token, one 200704 request costs
# 1.546e9 B, and KV_BYTES 3.15e9 therefore gives 2.04x. Shorter requests
# use the spare blocks; a third LONG request queues.
# conversations: measured live, KV_BYTES 3.15e9/rank reports 398,658
# tokens = 1.99x for a 200704-token request (7,892 B/token at THIS shape;
# the per-token price is shape-dependent, 7,701 at 262144). Shorter
# requests use the spare blocks; a third LONG request queues.
# The arena is sized by max_model_len, NOT by KV_BYTES: the planner
# reserves registry 9190 B/token x max_model_len (2.48 GiB at 262144 ->
# arena 6,248 slots; 1.91 GiB at 200704 -> 6,496). The pool cut buys
@@ -193,8 +194,9 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
}
@test "KV dtype is fp8 — the incumbent, and the only one that boots" {
# fp8 costs 7,701 B/token, decomposed as 6,138 attention + 1,563
# mamba/side (dtype-independent); bf16 costs 13,839 (12,276 + 1,563).
# fp8 costs ~7,701 B/token at 262144 (7,892 at 200704 — shape-dependent),
# decomposed as 6,138 attention + 1,563 mamba/side (dtype-independent);
# bf16 costs 13,839 (12,276 + 1,563).
# bf16 was abandoned 2026-10-01: its 2 x 262144 boot died with 0 bytes
# free on a real request, and the SEQS=2 slot count it was designed
# around cannot compile this model at all (see the shape test above).