diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index 4457cc3..ed44f86 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -277,7 +277,9 @@ Three instruments the PromQL above cannot reach: (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high - prefill rate is not always a client that sends new tokens. + prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` - + exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner: + [[frank-archon]]. - **Bucket the arriving request too** — `histogram_quantile` over `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees sequences that are already running; the one waiting for its first token pays the same steps and diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index a6d7f2f..855c2e7 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -349,14 +349,23 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 % replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`, - `core/sched/scheduler.cpp:953-978`), and the default `checkpoint policy geometric-backoff, base - 2048` thins the snapshots deep in a prompt, so a hit at 150k can resume at 128k. The cause object - agrees — every `prefix_hit_cuts_total` carries `reason="no_snapshot"` (`windowed_group` and - `adopt_failed` are dead) — but the event count is small and the tokens are the cost, so read - `hit_cut_tokens`. An arm, not an edit: - `--checkpoint-policy floor` (or a shorter `--checkpoint-interval`) buys the replay back with - pinned-RAM checkpoint slots, and those slots come out of the same pinned host memory the expert - pool claims — a resolver fight, and `prefix_hit_cut_tokens_total` is the meter. + `core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about + `log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as + deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840 + · … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every + `prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the + cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly + **`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs + retention only - the writes happen at every interval whatever the policy says (only `--prefill-span` + skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM + alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB + against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` - + the pinning room left beside the expert pool at boot - as the hard bound the ledger line names. + Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a + chain that keeps resuming at one deep point stops paying for it. Shortening + `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest + `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at + interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes. - **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly), declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows