frank-primary: record the measured shape-D numbers (398,658 tokens, 1.99x, 6,496 slots)

Comments/docs only - no functional change. The live boot of c769ff2 reported:
  GPU KV cache size: 398,658 tokens,
  Maximum concurrency for 200,704 tokens per request: 1.99x
  expert cache space 14.87 GiB / 6,496 slots of 2.344 MiB

The arena prediction (6,496) landed exactly; the concurrency prediction (2.04x)
did not, because the per-token KV price is SHAPE-DEPENDENT: 7,701 B/token at
MAXLEN 262144 (6.1e9 -> 792,069 tokens) but 7,892 at 200704 (3.146e9 ->
398,658). Never extrapolate a concurrency figure across geometries.

Also smoke-verified: three concurrent 67-token requests answered in parallel,
~4.1 s each - the 3 slots are real, and the pool admits 2 full-length.
This commit is contained in:
Frank
2026-10-02 11:48:09 -07:00
parent c769ff2f63
commit 295d5784d3
5 changed files with 39 additions and 28 deletions
+9 -5
View File
@@ -32,11 +32,15 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
# --- qualified engine shape (2026-10-02, shape D) ---------------------------
# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
# (SEQS 3), but the POOL, not the slot count, decides how many full-length
# conversations are resident. fp8 measures 7,701 B/token (6,138 attention +
# 1,563 mamba/side; measured: 6,100,000,000 B -> 792,069 tokens), so one
# 200704-token request costs 1.546e9 B. KV_BYTES 3.15e9 admits 2.04 such
# requests; shorter requests use the spare blocks, and a third LONG request
# queues rather than being refused.
# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
# GPU KV cache size: 398,658 tokens,
# Maximum concurrency for 200,704 tokens per request: 1.99x
# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is
# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token
# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a
# concurrency figure across geometries. Shorter requests use the spare blocks,
# and a third LONG request queues rather than being refused. Verified: three
# concurrent 67-token requests answered in parallel, ~4.1 s each.
#
# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
# against the then-deployed known-good shape, production stopped for each):
+3 -2
View File
@@ -13,8 +13,9 @@
#
# Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
# conversations at ~2 full-length (2.04x); shorter requests use the spare
# blocks. --max-num-seqs stays 3 because SEQS=2 CANNOT boot this model
# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
# CANNOT boot this model
# (torch.fx ConstraintViolationError on query_start_loc, measured at both
# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). NO speculative
# decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry