command-center/agent/managed-skills/radiance-decode-prefill-interference-census/SKILL.md

name, description

name	description
radiance-decode-prefill-interference-census	Diagnose decode collapsing while another session prefills on a radiance endpoint: phase-ratio step counters, lag test, metric-prefix and scrape traps, and which knob acts on the measured mechanism.

Read core/config.cpp (help table + validator) in the PINNED image source before writing any radiance flag, enum member, or metric name into a runbook: config_parse refuses unknown enum values at boot, so a remembered name can make the recorded arm undeployable. Enumerate members from the validator's error string, never from memory.

Run the census on the pinned build's own metric set. An older pin ranks the wrong mechanism: on radiance 1.0.13 the bucket table put a "collapse" on a shape whose counters did not exist yet, because every counter resets at a restart and the metric set changes with the build. Prove the window is one shape (up, or the first sample of a counter only the current build exports) before ranking buckets.

Step ledger first, knobs second

Three counters decide which mechanism you are looking at, and none of them is a rate:

• radiance:engine_step_seconds_decode_total / engine_steps_decode_total = mean decode-only step.
• (engine_step_seconds_total - engine_step_seconds_decode_total) / (engine_steps_total - engine_steps_decode_total) = mean chunk step (a chunk alone, or a chunk with decode rows beside it).
• engine_steps_held_total — if it is zero, nothing is protecting decoders.

If decode-only steps stay ~15 ms while the rate collapses, the mechanism is step-time sharing (a decode row in a chunk step waits the chunk's whole time), and any eviction/residency story is worth ~10 % of one step. The residency half is not censusable: the arena / expert slab / heat engine / prefill stager have no metric at all — they are read from the boot ledger, /stats, and /diagnostics.

Bucket, then lag-test, before believing a cause

Bucket radiance:decode_tokens_per_second by radiance:prefill_tokens_per_second from the same scrape, step ≤ 30 s (a 15 s scrape against a 120 s step samples one in eight). Per-stream rate = decode gauge ÷ vllm:num_requests_running; the tps gauges are ~1 s decayed batch-pooled windows.

Then shift the prefill series by lags and re-bucket:

• lag-0-only ⇒ step-time sharing — the chunk is holding the device.
• a lagged tail (2–14 min) ⇒ something evicted (arena loan, expert slab, KV).
• bursts in vllm:num_preemptions_total ⇒ a preempted sequence stopped and is re-prefilling from token 0: a stall, not sludge.

Also bucket TTFT (histogram_quantile over vllm:time_to_first_token_seconds_bucket) by the same series: the decode gauge only sees sequences already running, so an arriving request pays the same steps and shows in no decode number.

Metric traps

• Latency is vllm:, cadence is radiance:. TTFT / inter-token / per-output-token / e2e / decode-time histograms and the running/waiting gauges exist only under the vLLM compatibility prefix. Preemption is vllm:num_preemptions_total (counter); radiance:num_requests_preempted is the instant depth. A radiance:-only query finds no latency and no preemption and reads as "the endpoint never stalls anyone".
• A 2-minute mean step time dilutes the chunk — if ~5 % of steps carry a third of the step time, step_seconds/steps cannot move. Use the phase ratio.
• prefix_hit_cut_tokens_total is prefill load you created yourself: on a hybrid (linear + paged) model a hit reuses only to the last linear-state checkpoint, so cached tokens replay as prefill. A high prefill rate is not always a client sending new tokens.
• Radiance's own step-detail histograms (prefill_steps_duration, prefill_step_tokens, token_queue_wait) may be exported but unscraped; per-request forensics then mean curl localhost:7770/stats on the endpoint's host, and /diagnostics answers "is the box the limit" in one call (configuration drift, VRAM split, per-card link chain and clocks, findings with fixes).
• Rootless container logs: read as the deploy user (podman logs <svc>), never sudo podman logs; check where the serve script redirects before trusting a log file.
• In a temporary/remote shell, zsh aborts on a non-matching glob — quote or guard /sys/... wildcards.

Knob semantics (radiance 1.8.x, GUIDE §4.5; pinned source is referee)

• --prefill-interleave N — the decoder dial: N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing. Costs the prefilling prompt's TTFT and a queued request its first step. Counts as engine_steps_held_total.
• --busy-prefill-chunk — a prefill-speed dial, not decoder protection: auto reserves 1+spec rows per decoding sequence and 1/8 of the budget per waiting request, and the source refuses to halve for decode rows alone. Never below the prefill stager's smallest pass (named on the boot line) — below it every routed layer reads its pooled experts synchronously.
• --prefill-span N — needs --max-num-batched-tokens = N intervals; re-funds a lone prefill only, so it cannot buy back held steps while decoders run, and its extra intervals land on the arena that the expert slab loses.
• --checkpoint-policy — retention only, three names (geometric-backoff | tip-only | keep-all); snapshot writes happen at every interval whatever the policy says (only --prefill-span skips them), so keep-all buys a bounded replay for pinned host RAM and no extra PCIe. Shortening --checkpoint-interval does NOT fix deep divergence: the geometric ladder's coarsest scale is the largest interval × 2^k under the depth, so the deepest gap stays ≈ half the prompt while every cut multiplies the writes.
• --checkpoint-slots is a total, auto = policy held() + 1, per --max-num-seqs, within an eighth of host RAM; --max-num-batched-tokens is the blunt rung — it raises the arena the slab loses and its resonance on an expert-tiered payload can be violent.

Verify an arm, don't assume it

Flags may be passed conditionally by the serve script, so prove the flag reached argv (tr '\0' ' ' < /proc/$(pidof <engine>)/cmdline), then the counter the knob moves (engine_steps_held_total must leave zero), then the bucket pair. A deploy is not evidence a knob engaged. Rollback for each of these is its env line, not a rebuild.