Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ca1475543d | ||
|
|
f83c241668 | ||
|
|
b4d7f6e7fe |
@@ -100,18 +100,22 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
|
||||
| TTM gate | `TTM_MIN_MIB=197000`, `TTM_TARGET_MIB=224000` |
|
||||
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
|
||||
|
||||
Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
|
||||
the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
|
||||
`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
|
||||
The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
|
||||
sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
|
||||
per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
|
||||
`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
|
||||
host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
|
||||
(= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
|
||||
(≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
|
||||
directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
|
||||
the bigger cache, paid over PCIe when those experts are hit.
|
||||
Boot ledger for the live shape (2026-10-29, pin 1.8.1, `MAX_SEQS=4`,
|
||||
`HOST_POOL_MIB=66560`, `MAX_BATCHED_TOKENS=2048`): the resolver prints `elastic 26.73 GiB` →
|
||||
**experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**, `activation arena 990.36 MiB` (the
|
||||
price of the 2,048-token step: `256 tokens use 155.60 MiB, 1024 tokens use 472.86 MiB of
|
||||
896.20 MiB`), `prefill staging 373.65 MiB`, `vram_weights 24.30 GiB`, `host_pool 65.59 GiB`,
|
||||
`unclaimed 96.89 MiB`; referee line `kv: the pool backs 93% of the worst case (4 sequences all
|
||||
at 200000 tokens), 3.7 full-length sessions`; `device: the 2 cards reach each other's memory
|
||||
directly (peer to peer)`; sampler **12 rows** (= `seqs × (1 + speculative depth)`), 4 hardware
|
||||
queues a card (133.45 MiB), `already held 256.67 MiB`, mover `17519 slab slots, 7624 pinned
|
||||
pool slots, 3.58 GiB static vram`, `load: rank 0, mover took 24.30 GiB`.
|
||||
|
||||
The 2026-10-20 boot at `HOST_POOL_MIB=49152` printed the **same** `elastic 26.73 GiB /
|
||||
experts 20.72 GiB / kv cache 5.64 GiB / vram_weights 24.30 GiB`. The pool raise bought the
|
||||
endpoint no cache: the engine opened that pool with `0 B used` and the VRAM split never moved
|
||||
(Laws: the lever saturates at the expert floor). The era's own ledger line is superseded by
|
||||
this one; the number the old era quoted is gone with it.
|
||||
|
||||
The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
|
||||
anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
|
||||
@@ -244,7 +248,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
|
||||
| 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
|
||||
| 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
|
||||
| 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
|
||||
| 4 × 200,000, host pool 49,152 | 800,000 tok | 5.64 GiB = 747,492 tok | 93 % |
|
||||
| 4 × 200,000, host pool 66,560 (live) | 800,000 tok | **5.64 GiB — byte-identical to 49,152** | 93 % |
|
||||
|
||||
The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
|
||||
`ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
|
||||
|
||||
Reference in new issue
Block a user