docs(vault): answer "what are we doing wrong" on decode-vs-prefill

frank-archon Laws: add the measured interference census (decode-only step
15.5 ms vs 166 ms chunk steps; 4.7% of steps hold 34% of step time; median
decode 166 -> 13 t/s above 5k prefill tok/s, per-stream 6.7 t/s = one token
per shared step; TTFT p90 2.4 -> 18.0 s for arriving requests), the lag test
that kills the residency theory, the busy_cap arithmetic, and the correction
that --prefill-interleave is the decoder dial and auto is not.

Facts that change how an operator reads this box: /diagnostics exists and
reports link-below-max at 100% busy; latency metrics live under vllm:, cadence
under radiance:; preemption is vllm:num_preemptions_total (264 this boot, all
bursts) while the radiance gauge is instant depth; both root ports are x8 Gen5
max by board (31.5 GB/s ceiling, not 64); DDR is back at 4800 MT/s while the
4600 corruption fix is recorded -- verify before blaming garbage tokens.

§9 (LLM Performance Diagnosis): method corrected to the phase-ratio counters,
dead 1.0.13 numbers destroyed, four PromQL traps added (metric prefix, 1 s EMA
gauges vs 15 s scrape, counter resets across builds, 2-minute mean diluting the
chunk), instruments that cannot be reached from Prometheus, and the knob list
priced from the pinned 1.8.1 source instead of the stale guide.

serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md

@@ -239,45 +239,90 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp
 sequence stops producing entirely and then re-prefills from token 0, which is a
 stall, not sludge.
 
-Knobs, in the order worth trying them (radiance GUIDE §4.5):
+**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are
+looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is
+the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) /
+(steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with
+decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**,
+with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only
+14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so
+a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is
+worth ~10 % of one step — not a lever.
 
-- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N
-  steps carry decode rows only and admit nothing. Decoders keep their cadence; the
-  prompt pays that in TTFT. Counts as `engine_steps_held_total`.
-- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work:
-  `auto` (default) reserves a row plus the draft window per decode row behind it and
-  an eighth of the budget per waiting prefill/request, `off` lets the chunk take the
-  whole step (admission never runs while it does), `N` pins the share. Never under
-  the prefill stager's smallest pass — the startup line names it, and below it every
-  routed layer reads its pooled experts synchronously, which costs the prefill far
-  more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
-- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own
-  curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE
-  at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of
-  magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses
-  expert residency to the arena.
-- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every
-  shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena
-  back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing
-  at short prompts, a large TTFT gain at the far end of the window.
+**Four traps in the PromQL above.** Learn them before trusting a bucket:
+- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e
+  histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption
+  is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the
+  *instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the
+  endpoint never stalls anyone".
+- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is
+  `decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one
+  scrape in eight — use step ≤ 30 s for a bucket table.
+- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is
+  one shape (`up`, or the first sample of a counter that only the current build exports) before
+  ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse"
+  turned out to belong entirely to the older one.
+- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query
+  above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode.
 
-Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
-expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the
-decode rows inside that step also read experts over the link — a smaller shared budget helps twice.
-And **re-check
-`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
-pool are the same memory, and the failure looks like turn two taking as long as turn
-one.
+Three instruments the PromQL above cannot reach:
 
-Availability is a version question, not a config question: `--busy-prefill-chunk` and
-`--prefill-interleave` are absent from older pins — prove it on the running container
-(`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`), never from the vendored
-tree, and check the source clone's depth before dating a flag from `git log -S`
-(a `--depth 1` clone reports its own tip for every search).
+- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
+  `token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine /
+  prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on
+  the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry,
+  and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call:
+  configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes.
+- **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid
+  (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so
+  cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its
+  prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high
+  prefill rate is not always a client that sends new tokens.
+- **Bucket the arriving request too** — `histogram_quantile` over
+  `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees
+  sequences that are already running; the one waiting for its first token pays the same steps and
+  shows in no decode number.
 
-Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 =
-interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict
-for this fleet's shape are that page's.
+Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee):
+
+- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode
+  rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their
+  cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps.
+  Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone.
+- `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed
+  setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth
+  of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows
+  alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from
+  `engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill
+  stager's smallest pass — the largest arena level that lends, plus one, named on the startup line
+  `prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it
+  every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four
+  decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the
+  chunk blindly buys latency in both directions.
+- `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then
+  carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own
+  measurement says a span starts to win), while **every other step is planned at the shared
+  budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk
+  knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV
+  pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first.
+- `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the
+  arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses
+  decode — [[frank-archon]]).
+
+Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is
+reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a
+quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller
+slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the
+arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two
+taking exactly as long as turn one.
+
+Availability is a version question, not a config question: `--busy-prefill-chunk`,
+`--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running
+container (`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s
+`configuration` block, never from a vendored tree, and check a source clone's depth before dating a
+flag from `git log -S` (a `--depth 1` clone reports its own tip for every search).
+
+Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms.
 
 ## Related