frank-primary: record the measured shape-D numbers (398,658 tokens, 1.99x, 6,496 slots)

Comments/docs only - no functional change. The live boot of c769ff2 reported:
  GPU KV cache size: 398,658 tokens,
  Maximum concurrency for 200,704 tokens per request: 1.99x
  expert cache space 14.87 GiB / 6,496 slots of 2.344 MiB

The arena prediction (6,496) landed exactly; the concurrency prediction (2.04x)
did not, because the per-token KV price is SHAPE-DEPENDENT: 7,701 B/token at
MAXLEN 262144 (6.1e9 -> 792,069 tokens) but 7,892 at 200704 (3.146e9 ->
398,658). Never extrapolate a concurrency figure across geometries.

Also smoke-verified: three concurrent 67-token requests answered in parallel,
~4.1 s each - the 3 slots are real, and the pool admits 2 full-length.
This commit is contained in:
Frank committed 2026-10-02 11:48:09 -07:00
1 parent c769ff2f63
commit 295d5784d3
5 files changed
+39 -28

No files matched your search

+3 -2
View File
@@ -13,8 +13,9 @@
#
# Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
# conversations at ~2 full-length (2.04x); shorter requests use the spare
# blocks. --max-num-seqs stays 3 because SEQS=2 CANNOT boot this model
# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
# CANNOT boot this model
# (torch.fx ConstraintViolationError on query_start_loc, measured at both
# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). NO speculative
# decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry