docs(LLM): census the decode stall a prefill causes, and the knobs that turn it
Live step-shape census on frank-archon (radiance 1.0.13, boot of 2026-10-15): decode-only steps run 18.7 ms, prefill+mixed steps 317 ms — 60 % of engine wall-clock inside 8 % of the steps — and a day of Prometheus shows decode at 13.3 t/s median while prefill runs >=1000 t/s against 106-240 t/s when it does not. The lag test says the penalty is confined to the prefill window: no lingering expert-residency tail, and preemptions are zero. Adds the method as LLM Performance Diagnosis S9 (counter arithmetic, the lagged bucket correlation that separates step contention from residency churn, and the four radiance knobs with their published costs), and the endpoint's own numbers plus the verdict as two frank-archon laws: --busy-prefill-chunk and --prefill-interleave are absent from this pin (proven via the running container's --help), so MNBT is the only crank here and an image bump is gated by the 1.2.0 numerics dead end. No behavior change; docs only.
This commit is contained in:
1 parent
b9d21ae947
commit
e3a02e25c7
2 files changed
+92
-2
No files matched your search
@@ -234,6 +234,29 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
|
||||
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
|
||||
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
|
||||
GiB, refused).
|
||||
- **Decode pays for the chunk it shares a step with.** Census of the live engine
|
||||
(radiance 1.0.13, `/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
|
||||
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
|
||||
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
|
||||
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
|
||||
decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
|
||||
samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
|
||||
`prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
|
||||
bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
|
||||
0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
|
||||
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
|
||||
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
|
||||
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
|
||||
[[LLM Performance Diagnosis]] §9.
|
||||
- **The knobs for it are not in this pin.** `--busy-prefill-chunk` and
|
||||
`--prefill-interleave` (radiance GUIDE §4.5 — a shared chunk's cap, and N decode-only
|
||||
steps after each chunk) are absent from 1.0.13: proven on the running container
|
||||
(`podman exec frank-archon radiance --help` prints no `prefill`/`chunk` flag), and the
|
||||
2026-10-18 source tree carries both. The only crank this pin has is MNBT, and that is
|
||||
an arm and not an edit: the vendor prices a 1024 chunk at −19 % prefill to halve the
|
||||
stall, and this page's own MNBT law already knows 2560 collapses decode (MTP
|
||||
resonance). An image bump is gated by the 1.2.0 numerics dead end below, so any try
|
||||
re-scores draft acceptance before it is trusted.
|
||||
- **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it
|
||||
adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs ×
|
||||
(1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step
|
||||
|
||||
Reference in new issue
Block a user