From e3a02e25c7beaacf57d632aee37e775ca7eff670 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 04:11:20 -0700 Subject: [PATCH] docs(LLM): census the decode stall a prefill causes, and the knobs that turn it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live step-shape census on frank-archon (radiance 1.0.13, boot of 2026-10-15): decode-only steps run 18.7 ms, prefill+mixed steps 317 ms — 60 % of engine wall-clock inside 8 % of the steps — and a day of Prometheus shows decode at 13.3 t/s median while prefill runs >=1000 t/s against 106-240 t/s when it does not. The lag test says the penalty is confined to the prefill window: no lingering expert-residency tail, and preemptions are zero. Adds the method as LLM Performance Diagnosis S9 (counter arithmetic, the lagged bucket correlation that separates step contention from residency churn, and the four radiance knobs with their published costs), and the endpoint's own numbers plus the verdict as two frank-archon laws: --busy-prefill-chunk and --prefill-interleave are absent from this pin (proven via the running container's --help), so MNBT is the only crank here and an image bump is gated by the 1.2.0 numerics dead end. No behavior change; docs only. --- .../LLM/LLM Performance Diagnosis.md | 71 ++++++++++++++++++- serverAppConfigs-docs/LLM/frank-archon.md | 23 ++++++ 2 files changed, 92 insertions(+), 2 deletions(-) diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index 27a0e17..c032391 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -1,9 +1,9 @@ --- type: reference title: LLM Performance Diagnosis -description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing. +description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution, prefix-cache-aware routing, and the step-shape census behind decode stalls under a prefill. tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry] -timestamp: 2026-10-05T00:00:00-07:00 +timestamp: 2026-10-19T00:00:00-07:00 --- # LLM Performance Diagnosis @@ -213,9 +213,76 @@ function directly with synthetic bodies (stdlib-only policy module); conversation identity is the FIRST N KiB of prompt — a shorter tested prefix hashes differently per turn and pinning silently never happens. +## 9. Decode cadence under a prefill (the step-shape census) + +"Decode turns to sludge the moment a long prompt arrives, then recovers" is not a +link or a gather story — on a chunked-prefill engine the unit of interference is +**the step**. A step that carries a prefill chunk takes the chunk's time, and every +decode row sharing that step waits that long for its next token. Chunk size is +`min(remaining, --checkpoint-interval, --max-num-batched-tokens)`, so on a hybrid +model the interval caps it before the budget does. + +Census it without touching the model — the engine's own counters (`/stats`, +`/metrics`, cumulative since boot): + +``` +decode-only step = step_ns_decode / steps_decode +shared step = (step_ns - step_ns_decode) / (steps_prefill + steps_mixed) +``` + +Then bucket a day of `decode_tokens_per_second` by `prefill_tokens_per_second` +(Prometheus `query_range`, step 120 s) and **check the lag**: correlate decode with +prefill at lag 0 and again at 2-14 minutes. Lag-0-only ⇒ step-time contention, the +chunk is holding the device. A lagged tail ⇒ residency churn — the prefill's working +set evicted something decode needs (expert slab, KV) and decode pays after the +prefill is over. `preemptions` separates the third shape: a preempted-and-recomputed +sequence stops producing entirely and then re-prefills from token 0, which is a +stall, not sludge. + +Knobs, in the order worth trying them (radiance GUIDE §4.5): + +- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N + steps carry decode rows only and admit nothing. Decoders keep their cadence; the + prompt pays that in TTFT. Counts as `engine_steps_held_total`. +- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work: + `auto` (default) reserves a row plus the draft window per decode row behind it and + an eighth of the budget per waiting prefill/request, `off` lets the chunk take the + whole step (admission never runs while it does), `N` pins the share. Never under + the prefill stager's smallest pass — the startup line names it, and below it every + routed layer reads its pooled experts synchronously, which costs the prefill far + more than sharing saves. Counts as `engine_prefill_chunks_capped_total`. +- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The + vendor's curve on a 48-layer MoE at 200k on two cards: chunk 512 → 1750 t/s, 1024 + → 2200, 2048 → 2706 (a 185k prompt: 138 s against 91 s), so halving the stall + costs ~19 % of prefill. Raising it past the interval buys nothing and loses expert + residency to the arena. +- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step + while every shared step is planned at a smaller shared budget, which on an + expert-tiered box lends that arena back to the expert slab. Vendor TP2 + expert-tiered TTFT deltas vs a plain 2048 budget: −6.1 % at 2k, +2.3 % at 18k, + +49.1 % at 250k — long prompts only. + +Two traps. **The arena/slab coupling makes a shared step doubly expensive on an +expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would +have had (320 MiB/card at TP2 for a 4096 budget), so the decode rows inside that step +also read experts over the link — a smaller shared budget helps twice. And **re-check +`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV +pool are the same memory, and the failure looks like turn two taking as long as turn +one. + +Availability is a version question, not a config question: `--busy-prefill-chunk` and +`--prefill-interleave` are absent from older pins — prove it on the running container +(`podman exec --help | grep -iE "prefill|chunk"`), never from the vendored +tree, and check the source clone's depth before dating a flag from `git log -S` +(a `--depth 1` clone reports its own tip for every search). + +Run on: [[frank-archon]] (radiance 1.0.13, TP2 `expert_tiered`, budget 2048 = interval +2048) — the measured census and the verdict for this fleet's shape are that page's. + ## Related - [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law - [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling - [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms - [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound +- [[frank-archon]] — the endpoint where §9's step-shape census was run diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index 8f01e08..0e1d596 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -234,6 +234,29 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3 GiB, refused). +- **Decode pays for the chunk it shares a step with.** Census of the live engine + (radiance 1.0.13, `/stats` + `/metrics`, cumulative over the 2026-10-15 boot): + 1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps + at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the + chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every + decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721 + samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where + `prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets; + bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at + 0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a + busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual + confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute + resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method: + [[LLM Performance Diagnosis]] §9. +- **The knobs for it are not in this pin.** `--busy-prefill-chunk` and + `--prefill-interleave` (radiance GUIDE §4.5 — a shared chunk's cap, and N decode-only + steps after each chunk) are absent from 1.0.13: proven on the running container + (`podman exec frank-archon radiance --help` prints no `prefill`/`chunk` flag), and the + 2026-10-18 source tree carries both. The only crank this pin has is MNBT, and that is + an arm and not an edit: the vendor prices a 1024 chunk at −19 % prefill to halve the + stall, and this page's own MNBT law already knows 2560 collapses decode (MTP + resonance). An image bump is gated by the 1.2.0 numerics dead end below, so any try + re-scores draft acceptance before it is trusted. - **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step