Merge origin/main into docs/radiance-prefill-decode-census
Two conflicts, both from the 2026-10-19 vault compaction on main: * frontmatter description — main shortened it to satisfy OKF DESC (max 120 chars); this branch's longer wording would have failed the seam. Kept main's shape and folded the new §9 in by naming step-shape attribution instead of profiler attribution. * the Related bullet for [[vLLM Porting and Compile Triage]] — that page was destroyed on main, so the link is dropped rather than resurrected; this branch's [[frank-archon]] row is kept. Index rows regenerated with scripts/gen-vault-index.py.
This commit is contained in:
commit
e133d2412d
164 files changed
+3435
-15287
No files matched your search
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: LLM Performance Diagnosis
|
||||
description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution, prefix-cache-aware routing, and the step-shape census behind decode stalls under a prefill.
|
||||
description: "Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and step-shape attribution."
|
||||
tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
|
||||
timestamp: 2026-10-19T00:00:00-07:00
|
||||
---
|
||||
@@ -21,7 +21,7 @@ the cost; the problem is per-token expert movement and KV/attention tuning is
|
||||
ruled out entirely. A genuinely context-bound server shows a clear downward
|
||||
slope. Record cold vs warm separately — a small gap means the cache is
|
||||
working. The canonical flat run (fp8 expert-offload shape, a few hundred to
|
||||
sixteen thousand tokens of context) is in [[Tcclaviger Fork Ops]]'s profile.
|
||||
sixteen thousand tokens of context) is in 's profile.
|
||||
|
||||
## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
|
||||
|
||||
@@ -86,7 +86,7 @@ Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
|
||||
|
||||
No amount of link arithmetic would have produced those two conclusions: the
|
||||
suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
|
||||
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
|
||||
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It
|
||||
predicted a negative result in advance (spec decode cannot help a shape whose
|
||||
per-token cost scales with the token count) which matched an unexplained flat
|
||||
MTP measurement.
|
||||
@@ -117,7 +117,7 @@ per-request telemetry — it measures the real workload and costs nothing.
|
||||
- One `max_tokens=200` request is production traffic and fine; anything that
|
||||
pins the card is a bench → stop-and-mask first. Check WHICH instance you
|
||||
are reading (`systemctl status | head -12`; task ids jump across
|
||||
restarts). The engine's own ledger (`KV self size`, `n_slots`,
|
||||
restarts). The engine's own ledger (`KV self size`, `n_slots`
|
||||
`n_ctx_slot`) outranks your model of it.
|
||||
|
||||
## 5. The fork's data path (before blaming the link)
|
||||
@@ -130,7 +130,7 @@ tokens, so the step can silently cross the threshold onto the slow path as
|
||||
draft depth grows. Check the threshold's premise: a default tuned at
|
||||
12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
|
||||
several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
|
||||
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
|
||||
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required)
|
||||
check for the skip line before concluding autotuning ran. Gather-geometry
|
||||
knobs read at import time → each arm is a full reload.
|
||||
|
||||
@@ -189,7 +189,7 @@ analyze; `--analyze-only`).
|
||||
Measure with a **sequential-turn client**, not only a batch benchmark — a
|
||||
batch of distinct prompts parallelises prefill across replicas and hides the
|
||||
problem. The two failure modes: round-robin loses every other cache hit
|
||||
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
|
||||
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned)
|
||||
pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
|
||||
pinned). **The rule that gets both: route by request shape** — a request
|
||||
carrying more than one message is a continuation (agent loop re-sending its
|
||||
@@ -209,7 +209,7 @@ is not a per-request verdict (probe directly: same prompt twice to an idle
|
||||
replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
|
||||
`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
|
||||
against the live endpoint before dashboards. Test shape: drive the routing
|
||||
function directly with synthetic bodies (stdlib-only policy module);
|
||||
function directly with synthetic bodies (stdlib-only policy module)
|
||||
conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
|
||||
hashes differently per turn and pinning silently never happens.
|
||||
|
||||
@@ -281,8 +281,7 @@ Run on: [[frank-archon]] (radiance 1.0.13, TP2 `expert_tiered`, budget 2048 = in
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
|
||||
- the window discipline and usage-counter law
|
||||
- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
|
||||
- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound
|
||||
- [[frank-archon]] — the endpoint where §9's step-shape census was run
|
||||
Reference in new issue
Block a user