docs(vault): replace an invented --checkpoint-policy name with the three that exist
The replay law named `--checkpoint-policy floor`; config.cpp admits exactly
geometric-backoff (default), tip-only, keep-all and refuses anything else at
boot, so the recorded arm could never be deployed. Replaced with the real
semantics, priced from core/mem/prefix.{h,cpp}: the default keeps a geometric
ladder (9 snapshots for a 200k session at interval 2048 -- exactly the live
ledger line -- so a turn diverging d behind the tip can resume at tip-3d:
diverge at 120k, replay 21.7k); retention is not the write path, so keep-all
buys a bounded replay for pinned host RAM alone (196 slots x 111.98 MiB = 21.4
GiB against the store's eighth-of-host budget of 31.5 GiB, hard bound = room);
and shortening --checkpoint-interval cannot fix the deep case -- the coarsest
ladder scale is interval x 2^k under the depth, so the deepest gap stays about
half the prompt at every interval. §9 keeps only the three-name locator.
This commit is contained in:
@@ -277,7 +277,9 @@ Three instruments the PromQL above cannot reach:
|
||||
(linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so
|
||||
cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its
|
||||
prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high
|
||||
prefill rate is not always a client that sends new tokens.
|
||||
prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` -
|
||||
exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner:
|
||||
[[frank-archon]].
|
||||
- **Bucket the arriving request too** — `histogram_quantile` over
|
||||
`vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees
|
||||
sequences that are already running; the one waiting for its first token pays the same steps and
|
||||
|
||||
@@ -349,14 +349,23 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 %
|
||||
replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last
|
||||
**linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`,
|
||||
`core/sched/scheduler.cpp:953-978`), and the default `checkpoint policy geometric-backoff, base
|
||||
2048` thins the snapshots deep in a prompt, so a hit at 150k can resume at 128k. The cause object
|
||||
agrees — every `prefix_hit_cuts_total` carries `reason="no_snapshot"` (`windowed_group` and
|
||||
`adopt_failed` are dead) — but the event count is small and the tokens are the cost, so read
|
||||
`hit_cut_tokens`. An arm, not an edit:
|
||||
`--checkpoint-policy floor` (or a shorter `--checkpoint-interval`) buys the replay back with
|
||||
pinned-RAM checkpoint slots, and those slots come out of the same pinned host memory the expert
|
||||
pool claims — a resolver fight, and `prefix_hit_cut_tokens_total` is the meter.
|
||||
`core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about
|
||||
`log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as
|
||||
deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840
|
||||
· … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every
|
||||
`prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the
|
||||
cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly
|
||||
**`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs
|
||||
retention only - the writes happen at every interval whatever the policy says (only `--prefill-span`
|
||||
skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM
|
||||
alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB
|
||||
against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` -
|
||||
the pinning room left beside the expert pool at boot - as the hard bound the ledger line names.
|
||||
Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a
|
||||
chain that keeps resuming at one deep point stops paying for it. Shortening
|
||||
`--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
|
||||
`interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
|
||||
interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
|
||||
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
|
||||
adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
|
||||
declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
|
||||
|
||||
Reference in New Issue
Block a user