docs(LLM): census the decode stall a prefill causes, and the knobs that turn it

Live step-shape census on frank-archon (radiance 1.0.13, boot of 2026-10-15):
decode-only steps run 18.7 ms, prefill+mixed steps 317 ms — 60 % of engine
wall-clock inside 8 % of the steps — and a day of Prometheus shows decode at
13.3 t/s median while prefill runs >=1000 t/s against 106-240 t/s when it does
not. The lag test says the penalty is confined to the prefill window: no
lingering expert-residency tail, and preemptions are zero.

Adds the method as LLM Performance Diagnosis S9 (counter arithmetic, the lagged
bucket correlation that separates step contention from residency churn, and the
four radiance knobs with their published costs), and the endpoint's own numbers
plus the verdict as two frank-archon laws: --busy-prefill-chunk and
--prefill-interleave are absent from this pin (proven via the running
container's --help), so MNBT is the only crank here and an image bump is gated
by the 1.2.0 numerics dead end.

No behavior change; docs only.
This commit is contained in:
Frank committed 2026-10-19 04:11:20 -07:00
1 parent b9d21ae947
commit e3a02e25c7
2 files changed
+92 -2

No files matched your search

+23
View File
@@ -234,6 +234,29 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
GiB, refused).
- **Decode pays for the chunk it shares a step with.** Census of the live engine
(radiance 1.0.13, `/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
`prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
[[LLM Performance Diagnosis]] §9.
- **The knobs for it are not in this pin.** `--busy-prefill-chunk` and
`--prefill-interleave` (radiance GUIDE §4.5 — a shared chunk's cap, and N decode-only
steps after each chunk) are absent from 1.0.13: proven on the running container
(`podman exec frank-archon radiance --help` prints no `prefill`/`chunk` flag), and the
2026-10-18 source tree carries both. The only crank this pin has is MNBT, and that is
an arm and not an edit: the vendor prices a 1024 chunk at −19 % prefill to halve the
stall, and this page's own MNBT law already knows 2560 collapses decode (MTP
resonance). An image bump is gated by the 1.2.0 numerics dead end below, so any try
re-scores draft acceptance before it is trusted.
- **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it
adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs ×
(1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step