diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index 9a493df..ca0647e 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -1,9 +1,9 @@ --- type: reference title: LLM Performance Diagnosis -description: "Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and profiler attribution." +description: "Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and step-shape attribution." tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry] -timestamp: 2026-10-05T00:00:00-07:00 +timestamp: 2026-10-19T00:00:00-07:00 --- # LLM Performance Diagnosis @@ -213,9 +213,75 @@ function directly with synthetic bodies (stdlib-only policy module) conversation identity is the FIRST N KiB of prompt — a shorter tested prefix hashes differently per turn and pinning silently never happens. +## 9. Decode cadence under a prefill (the step-shape census) + +"Decode turns to sludge the moment a long prompt arrives, then recovers" is not a +link or a gather story — on a chunked-prefill engine the unit of interference is +**the step**. A step that carries a prefill chunk takes the chunk's time, and every +decode row sharing that step waits that long for its next token. Chunk size is +`min(remaining, --checkpoint-interval, --max-num-batched-tokens)`, so on a hybrid +model the interval caps it before the budget does. + +Census it without touching the model — the engine's own counters (`/stats`, +`/metrics`, cumulative since boot): + +``` +decode-only step = step_ns_decode / steps_decode +shared step = (step_ns - step_ns_decode) / (steps_prefill + steps_mixed) +``` + +Then bucket a day of `decode_tokens_per_second` by `prefill_tokens_per_second` +(Prometheus `query_range`, step 120 s) and **check the lag**: correlate decode with +prefill at lag 0 and again at 2-14 minutes. Lag-0-only ⇒ step-time contention, the +chunk is holding the device. A lagged tail ⇒ residency churn — the prefill's working +set evicted something decode needs (expert slab, KV) and decode pays after the +prefill is over. `preemptions` separates the third shape: a preempted-and-recomputed +sequence stops producing entirely and then re-prefills from token 0, which is a +stall, not sludge. + +Knobs, in the order worth trying them (radiance GUIDE §4.5): + +- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N + steps carry decode rows only and admit nothing. Decoders keep their cadence; the + prompt pays that in TTFT. Counts as `engine_steps_held_total`. +- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work: + `auto` (default) reserves a row plus the draft window per decode row behind it and + an eighth of the budget per waiting prefill/request, `off` lets the chunk take the + whole step (admission never runs while it does), `N` pins the share. Never under + the prefill stager's smallest pass — the startup line names it, and below it every + routed layer reads its pooled experts synchronously, which costs the prefill far + more than sharing saves. Counts as `engine_prefill_chunks_capped_total`. +- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The + vendor's curve on a 48-layer MoE at 200k on two cards: chunk 512 → 1750 t/s, 1024 + → 2200, 2048 → 2706 (a 185k prompt: 138 s against 91 s), so halving the stall + costs ~19 % of prefill. Raising it past the interval buys nothing and loses expert + residency to the arena. +- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step + while every shared step is planned at a smaller shared budget, which on an + expert-tiered box lends that arena back to the expert slab. Vendor TP2 + expert-tiered TTFT deltas vs a plain 2048 budget: −6.1 % at 2k, +2.3 % at 18k, + +49.1 % at 250k — long prompts only. + +Two traps. **The arena/slab coupling makes a shared step doubly expensive on an +expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would +have had (320 MiB/card at TP2 for a 4096 budget), so the decode rows inside that step +also read experts over the link — a smaller shared budget helps twice. And **re-check +`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV +pool are the same memory, and the failure looks like turn two taking as long as turn +one. + +Availability is a version question, not a config question: `--busy-prefill-chunk` and +`--prefill-interleave` are absent from older pins — prove it on the running container +(`podman exec --help | grep -iE "prefill|chunk"`), never from the vendored +tree, and check the source clone's depth before dating a flag from `git log -S` +(a `--depth 1` clone reports its own tip for every search). + +Run on: [[frank-archon]] (radiance 1.0.13, TP2 `expert_tiered`, budget 2048 = interval +2048) — the measured census and the verdict for this fleet's shape are that page's. + ## Related - the window discipline and usage-counter law - [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling - [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms -- the compile fix for the pointwise-glue bound +- [[frank-archon]] — the endpoint where §9's step-shape census was run diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index ca81600..1dc6542 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -239,6 +239,29 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3 GiB, refused). +- **Decode pays for the chunk it shares a step with.** Census of the live engine + (radiance 1.0.13, `/stats` + `/metrics`, cumulative over the 2026-10-15 boot): + 1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps + at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the + chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every + decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721 + samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where + `prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets; + bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at + 0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a + busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual + confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute + resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method: + [[LLM Performance Diagnosis]] §9. +- **The knobs for it are not in this pin.** `--busy-prefill-chunk` and + `--prefill-interleave` (radiance GUIDE §4.5 — a shared chunk's cap, and N decode-only + steps after each chunk) are absent from 1.0.13: proven on the running container + (`podman exec frank-archon radiance --help` prints no `prefill`/`chunk` flag), and the + 2026-10-18 source tree carries both. The only crank this pin has is MNBT, and that is + an arm and not an edit: the vendor prices a 1024 chunk at −19 % prefill to halve the + stall, and this page's own MNBT law already knows 2560 collapses decode (MTP + resonance). An image bump is gated by the 1.2.0 numerics dead end below, so any try + re-scores draft acceptance before it is trusted. - **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 77cff4e..cb8d92d 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -35,7 +35,7 @@ the endpoint's **Eras** table, and the numbers a retired era measured are gone. | [[GGUF Compatibility]] | Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can. | | [[Heterogeneous Split Triage]] | Triage before splitting across unequal accelerators: link characterisation, real bottleneck, what placement never buys. | | [[Llama Cpp Serving Sizing]] | Measurement-first sizing for llama.cpp shapes: slot pools against a headroom band, per-device accounting, prompt cache. | -| [[LLM Performance Diagnosis]] | Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and profiler attribution. | +| [[LLM Performance Diagnosis]] | Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and step-shape attribution. | | [[Model Swap and Rollback]] | The model-swap discipline: fit preflight, fetch verification, build and flag checks, live validation, three-pin revert. | | [[PCIe Link and Atomics]] | Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives. | | [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. | diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 5cfdd04..145eb6b 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -154,7 +154,7 @@ is a locator: one row, one clause, no numbers. | [[GGUF Compatibility]] | Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can. | | [[Heterogeneous Split Triage]] | Triage before splitting across unequal accelerators: link characterisation, real bottleneck, what placement never buys. | | [[Llama Cpp Serving Sizing]] | Measurement-first sizing for llama.cpp shapes: slot pools against a headroom band, per-device accounting, prompt cache. | -| [[LLM Performance Diagnosis]] | Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and profiler attribution. | +| [[LLM Performance Diagnosis]] | Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and step-shape attribution. | | [[Model Swap and Rollback]] | The model-swap discipline: fit preflight, fetch verification, build and flag checks, live validation, three-pin revert. | | [[PCIe Link and Atomics]] | Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives. | | [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |