From b1fc1113c60172bc603539b41934244905da74d0 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 20 Oct 2026 22:34:13 -0700 Subject: [PATCH] docs(vault): replace an invented --checkpoint-policy name with the three that exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The replay law named `--checkpoint-policy floor`; config.cpp admits exactly geometric-backoff (default), tip-only, keep-all and refuses anything else at boot, so the recorded arm could never be deployed. Replaced with the real semantics, priced from core/mem/prefix.{h,cpp}: the default keeps a geometric ladder (9 snapshots for a 200k session at interval 2048 -- exactly the live ledger line -- so a turn diverging d behind the tip can resume at tip-3d: diverge at 120k, replay 21.7k); retention is not the write path, so keep-all buys a bounded replay for pinned host RAM alone (196 slots x 111.98 MiB = 21.4 GiB against the store's eighth-of-host budget of 31.5 GiB, hard bound = room); and shortening --checkpoint-interval cannot fix the deep case -- the coarsest ladder scale is interval x 2^k under the depth, so the deepest gap stays about half the prompt at every interval. §9 keeps only the three-name locator. --- .../LLM/LLM Performance Diagnosis.md | 4 ++- serverAppConfigs-docs/LLM/frank-archon.md | 25 +++++++++++++------ 2 files changed, 20 insertions(+), 9 deletions(-) diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index 4457cc3..ed44f86 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -277,7 +277,9 @@ Three instruments the PromQL above cannot reach: (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high - prefill rate is not always a client that sends new tokens. + prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` - + exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner: + [[frank-archon]]. - **Bucket the arriving request too** — `histogram_quantile` over `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees sequences that are already running; the one waiting for its first token pays the same steps and diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index a6d7f2f..855c2e7 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -349,14 +349,23 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 % replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`, - `core/sched/scheduler.cpp:953-978`), and the default `checkpoint policy geometric-backoff, base - 2048` thins the snapshots deep in a prompt, so a hit at 150k can resume at 128k. The cause object - agrees — every `prefix_hit_cuts_total` carries `reason="no_snapshot"` (`windowed_group` and - `adopt_failed` are dead) — but the event count is small and the tokens are the cost, so read - `hit_cut_tokens`. An arm, not an edit: - `--checkpoint-policy floor` (or a shorter `--checkpoint-interval`) buys the replay back with - pinned-RAM checkpoint slots, and those slots come out of the same pinned host memory the expert - pool claims — a resolver fight, and `prefix_hit_cut_tokens_total` is the meter. + `core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about + `log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as + deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840 + · … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every + `prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the + cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly + **`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs + retention only - the writes happen at every interval whatever the policy says (only `--prefill-span` + skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM + alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB + against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` - + the pinning room left beside the expert pool at boot - as the hard bound the ledger line names. + Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a + chain that keeps resuming at one deep point stops paying for it. Shortening + `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest + `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at + interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes. - **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly), declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows