Merge origin/main into docs/radiance-prefill-decode-census

Two conflicts, both from the 2026-10-19 vault compaction on main:

* frontmatter description — main shortened it to satisfy OKF DESC
  (max 120 chars); this branch's longer wording would have failed the
  seam. Kept main's shape and folded the new §9 in by naming
  step-shape attribution instead of profiler attribution.
* the Related bullet for [[vLLM Porting and Compile Triage]] — that
  page was destroyed on main, so the link is dropped rather than
  resurrected; this branch's [[frank-archon]] row is kept.

Index rows regenerated with scripts/gen-vault-index.py.
This commit is contained in:
Frank committed 2026-10-19 14:20:05 -07:00
commit e133d2412d
164 files changed
+3435 -15287

No files matched your search

@@ -1,7 +1,7 @@
---
type: reference
title: LLM Performance Diagnosis
description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution, prefix-cache-aware routing, and the step-shape census behind decode stalls under a prefill.
description: "Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and step-shape attribution."
tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
timestamp: 2026-10-19T00:00:00-07:00
---
@@ -21,7 +21,7 @@ the cost; the problem is per-token expert movement and KV/attention tuning is
ruled out entirely. A genuinely context-bound server shows a clear downward
slope. Record cold vs warm separately — a small gap means the cache is
working. The canonical flat run (fp8 expert-offload shape, a few hundred to
sixteen thousand tokens of context) is in [[Tcclaviger Fork Ops]]'s profile.
sixteen thousand tokens of context) is in 's profile.
## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
@@ -86,7 +86,7 @@ Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
No amount of link arithmetic would have produced those two conclusions: the
suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It
predicted a negative result in advance (spec decode cannot help a shape whose
per-token cost scales with the token count) which matched an unexplained flat
MTP measurement.
@@ -117,7 +117,7 @@ per-request telemetry — it measures the real workload and costs nothing.
- One `max_tokens=200` request is production traffic and fine; anything that
pins the card is a bench → stop-and-mask first. Check WHICH instance you
are reading (`systemctl status | head -12`; task ids jump across
restarts). The engine's own ledger (`KV self size`, `n_slots`,
restarts). The engine's own ledger (`KV self size`, `n_slots`
`n_ctx_slot`) outranks your model of it.
## 5. The fork's data path (before blaming the link)
@@ -130,7 +130,7 @@ tokens, so the step can silently cross the threshold onto the slow path as
draft depth grows. Check the threshold's premise: a default tuned at
12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required)
check for the skip line before concluding autotuning ran. Gather-geometry
knobs read at import time → each arm is a full reload.
@@ -189,7 +189,7 @@ analyze; `--analyze-only`).
Measure with a **sequential-turn client**, not only a batch benchmark — a
batch of distinct prompts parallelises prefill across replicas and hides the
problem. The two failure modes: round-robin loses every other cache hit
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned)
pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
pinned). **The rule that gets both: route by request shape** — a request
carrying more than one message is a continuation (agent loop re-sending its
@@ -209,7 +209,7 @@ is not a per-request verdict (probe directly: same prompt twice to an idle
replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
against the live endpoint before dashboards. Test shape: drive the routing
function directly with synthetic bodies (stdlib-only policy module);
function directly with synthetic bodies (stdlib-only policy module)
conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
hashes differently per turn and pinning silently never happens.
@@ -281,8 +281,7 @@ Run on: [[frank-archon]] (radiance 1.0.13, TP2 `expert_tiered`, budget 2048 = in
## Related
- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
- the window discipline and usage-counter law
- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound
- [[frank-archon]] — the endpoint where §9's step-shape census was run