frank-primary: record the measured shape-D numbers (398,658 tokens, 1.99x, 6,496 slots)
Comments/docs only - no functional change. The live boot of c769ff2 reported:
GPU KV cache size: 398,658 tokens,
Maximum concurrency for 200,704 tokens per request: 1.99x
expert cache space 14.87 GiB / 6,496 slots of 2.344 MiB
The arena prediction (6,496) landed exactly; the concurrency prediction (2.04x)
did not, because the per-token KV price is SHAPE-DEPENDENT: 7,701 B/token at
MAXLEN 262144 (6.1e9 -> 792,069 tokens) but 7,892 at 200704 (3.146e9 ->
398,658). Never extrapolate a concurrency figure across geometries.
Also smoke-verified: three concurrent 67-token requests answered in parallel,
~4.1 s each - the 3 slots are real, and the pool admits 2 full-length.
This commit is contained in:
@@ -32,11 +32,15 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
# --- qualified engine shape (2026-10-02, shape D) ---------------------------
|
||||
# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
|
||||
# (SEQS 3), but the POOL, not the slot count, decides how many full-length
|
||||
# conversations are resident. fp8 measures 7,701 B/token (6,138 attention +
|
||||
# 1,563 mamba/side; measured: 6,100,000,000 B -> 792,069 tokens), so one
|
||||
# 200704-token request costs 1.546e9 B. KV_BYTES 3.15e9 admits 2.04 such
|
||||
# requests; shorter requests use the spare blocks, and a third LONG request
|
||||
# queues rather than being refused.
|
||||
# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
|
||||
# GPU KV cache size: 398,658 tokens,
|
||||
# Maximum concurrency for 200,704 tokens per request: 1.99x
|
||||
# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is
|
||||
# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token
|
||||
# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a
|
||||
# concurrency figure across geometries. Shorter requests use the spare blocks,
|
||||
# and a third LONG request queues rather than being refused. Verified: three
|
||||
# concurrent 67-token requests answered in parallel, ~4.1 s each.
|
||||
#
|
||||
# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
|
||||
# against the then-deployed known-good shape, production stopped for each):
|
||||
|
||||
@@ -13,8 +13,9 @@
|
||||
#
|
||||
# Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
|
||||
# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
|
||||
# conversations at ~2 full-length (2.04x); shorter requests use the spare
|
||||
# blocks. --max-num-seqs stays 3 because SEQS=2 CANNOT boot this model
|
||||
# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
|
||||
# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
|
||||
# CANNOT boot this model
|
||||
# (torch.fx ConstraintViolationError on query_start_loc, measured at both
|
||||
# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). NO speculative
|
||||
# decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
|
||||
|
||||
Reference in New Issue
Block a user