frank-primary: record the measured shape-D numbers (398,658 tokens, 1.99x, 6,496 slots)
Comments/docs only - no functional change. The live boot of c769ff2 reported:
GPU KV cache size: 398,658 tokens,
Maximum concurrency for 200,704 tokens per request: 1.99x
expert cache space 14.87 GiB / 6,496 slots of 2.344 MiB
The arena prediction (6,496) landed exactly; the concurrency prediction (2.04x)
did not, because the per-token KV price is SHAPE-DEPENDENT: 7,701 B/token at
MAXLEN 262144 (6.1e9 -> 792,069 tokens) but 7,892 at 200704 (3.146e9 ->
398,658). Never extrapolate a concurrency figure across geometries.
Also smoke-verified: three concurrent 67-token requests answered in parallel,
~4.1 s each - the 3 slots are real, and the pool admits 2 full-length.
This commit is contained in:
1 parent
c769ff2f63
commit
295d5784d3
5 files changed
+39
-28
No files matched your search
@@ -280,7 +280,7 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=200704 200704-token window
|
||||
FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
|
||||
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,701 B/token -> 2.04x)
|
||||
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x)
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
@@ -288,12 +288,16 @@ NO --speculative-config, NO --expert-prefetch
|
||||
```
|
||||
|
||||
**The pool, not the slot count, sets how many conversations live at once.**
|
||||
fp8 costs 7,701 B/token (measured 6,138 attention + 1,563 mamba/side,
|
||||
dtype-independent), so one 200704-token request costs 1.546e9 B and KV_BYTES
|
||||
3.15e9 admits **2.04 full-length requests**. The endpoint advertises 3 slots: a
|
||||
third LONG request queues rather than being refused, and shorter requests use
|
||||
the spare blocks. Two contexts at 200704 are a capacity contract, not a
|
||||
throughput one (2-way concurrency summed BELOW parity at 4064 ctx).
|
||||
Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
|
||||
concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
|
||||
The per-token price is **shape-dependent**: the 262144 shape priced at 7,701
|
||||
B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer,
|
||||
so a concurrency figure never extrapolates across geometries. The endpoint
|
||||
advertises 3 slots: a third LONG request queues rather than being refused, and
|
||||
shorter requests use the spare blocks. Verified: three concurrent 67-token
|
||||
requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a
|
||||
capacity contract, not a throughput one (2-way concurrency summed BELOW parity
|
||||
at 4064 ctx).
|
||||
|
||||
#### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02)
|
||||
|
||||
@@ -346,12 +350,12 @@ stay up: it left 0 bytes free at runtime and a real client request (995-token
|
||||
prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
|
||||
... 0 bytes free`.
|
||||
|
||||
The correction is a pool cut plus a window trim, not a downgrade. fp8 costs
|
||||
7,701 B/token, so 3 x 200704 = 602,112 tokens would need 4.64e9 B for three
|
||||
full-length contexts; pinning the pool at 3.15e9 instead admits 2.04 of them.
|
||||
That is deliberate: the pool is the runtime-HEADROOM lever (the shape this
|
||||
replaced died with 0 bytes free), while the arena is sized by `max_model_len`
|
||||
alone. A third full-length request queues; short requests use the spare blocks.
|
||||
The correction is a pool cut plus a window trim, not a downgrade. Three
|
||||
full-length 200704-token contexts need ~4.75e9 B; pinning the pool at 3.15e9
|
||||
instead admits **1.99** of them. That is deliberate: the pool is the
|
||||
runtime-HEADROOM lever (the shape this replaced died with 0 bytes free), while
|
||||
the arena is sized by `max_model_len` alone. A third full-length request queues;
|
||||
short requests use the spare blocks.
|
||||
|
||||
Boot-probed 2026-10-02 (transient systemd units at the production uid +
|
||||
memlock, full-log capture; the rootless bench probe had masked the production
|
||||
@@ -364,7 +368,7 @@ memory environment). The night searched two concurrency contracts:
|
||||
| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
|
||||
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
|
||||
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
|
||||
| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 2.04x concurrency, arena 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
|
||||
| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
|
||||
|
||||
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
|
||||
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
|
||||
|
||||
Reference in new issue
Block a user