frank-primary: record the measured shape-D numbers (398,658 tokens, 1.99x, 6,496 slots)

Comments/docs only - no functional change. The live boot of c769ff2 reported:
  GPU KV cache size: 398,658 tokens,
  Maximum concurrency for 200,704 tokens per request: 1.99x
  expert cache space 14.87 GiB / 6,496 slots of 2.344 MiB

The arena prediction (6,496) landed exactly; the concurrency prediction (2.04x)
did not, because the per-token KV price is SHAPE-DEPENDENT: 7,701 B/token at
MAXLEN 262144 (6.1e9 -> 792,069 tokens) but 7,892 at 200704 (3.146e9 ->
398,658). Never extrapolate a concurrency figure across geometries.

Also smoke-verified: three concurrent 67-token requests answered in parallel,
~4.1 s each - the 3 slots are real, and the pool admits 2 full-length.
This commit is contained in:
Frank committed 2026-10-02 11:48:09 -07:00
1 parent c769ff2f63
commit 295d5784d3
5 files changed
+39 -28

No files matched your search

@@ -280,7 +280,7 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
```
FRANK_PRIMARY_MAXLEN=200704 200704-token window
FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,701 B/token -> 2.04x)
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
@@ -288,12 +288,16 @@ NO --speculative-config, NO --expert-prefetch
```
**The pool, not the slot count, sets how many conversations live at once.**
fp8 costs 7,701 B/token (measured 6,138 attention + 1,563 mamba/side,
dtype-independent), so one 200704-token request costs 1.546e9 B and KV_BYTES
3.15e9 admits **2.04 full-length requests**. The endpoint advertises 3 slots: a
third LONG request queues rather than being refused, and shorter requests use
the spare blocks. Two contexts at 200704 are a capacity contract, not a
throughput one (2-way concurrency summed BELOW parity at 4064 ctx).
Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
The per-token price is **shape-dependent**: the 262144 shape priced at 7,701
B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer,
so a concurrency figure never extrapolates across geometries. The endpoint
advertises 3 slots: a third LONG request queues rather than being refused, and
shorter requests use the spare blocks. Verified: three concurrent 67-token
requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a
capacity contract, not a throughput one (2-way concurrency summed BELOW parity
at 4064 ctx).
#### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02)
@@ -346,12 +350,12 @@ stay up: it left 0 bytes free at runtime and a real client request (995-token
prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
... 0 bytes free`.
The correction is a pool cut plus a window trim, not a downgrade. fp8 costs
7,701 B/token, so 3 x 200704 = 602,112 tokens would need 4.64e9 B for three
full-length contexts; pinning the pool at 3.15e9 instead admits 2.04 of them.
That is deliberate: the pool is the runtime-HEADROOM lever (the shape this
replaced died with 0 bytes free), while the arena is sized by `max_model_len`
alone. A third full-length request queues; short requests use the spare blocks.
The correction is a pool cut plus a window trim, not a downgrade. Three
full-length 200704-token contexts need ~4.75e9 B; pinning the pool at 3.15e9
instead admits **1.99** of them. That is deliberate: the pool is the
runtime-HEADROOM lever (the shape this replaced died with 0 bytes free), while
the arena is sized by `max_model_len` alone. A third full-length request queues;
short requests use the spare blocks.
Boot-probed 2026-10-02 (transient systemd units at the production uid +
memlock, full-log capture; the rootless bench probe had masked the production
@@ -364,7 +368,7 @@ memory environment). The night searched two concurrency contracts:
| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 2.04x concurrency, arena 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even