Frank
8af40c402b
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
52 lines
6.3 KiB
Markdown
52 lines
6.3 KiB
Markdown
---
|
||
name: radiance-decode-prefill-interference-census
|
||
description: "Diagnose decode collapsing while another session prefills on a radiance endpoint: phase-ratio step counters, lag test, metric-prefix and scrape traps, and which knob acts on the measured mechanism."
|
||
---
|
||
|
||
Read `core/config.cpp` (help table + validator) in the PINNED image source before writing any radiance flag, enum member, or metric name into a runbook: `config_parse` refuses unknown enum values at boot, so a remembered name can make the recorded arm undeployable. Enumerate members from the validator's error string, never from memory.
|
||
|
||
**Run the census on the pinned build's own metric set.** An older pin ranks the wrong mechanism: on radiance 1.0.13 the bucket table put a "collapse" on a shape whose counters did not exist yet, because every counter resets at a restart and the metric set changes with the build. Prove the window is one shape (`up`, or the first sample of a counter only the current build exports) before ranking buckets.
|
||
|
||
## Step ledger first, knobs second
|
||
|
||
Three counters decide which mechanism you are looking at, and none of them is a rate:
|
||
|
||
- `radiance:engine_step_seconds_decode_total / engine_steps_decode_total` = mean **decode-only** step.
|
||
- `(engine_step_seconds_total - engine_step_seconds_decode_total) / (engine_steps_total - engine_steps_decode_total)` = mean **chunk step** (a chunk alone, or a chunk with decode rows beside it).
|
||
- `engine_steps_held_total` — if it is zero, nothing is protecting decoders.
|
||
|
||
If decode-only steps stay ~15 ms while the rate collapses, the mechanism is **step-time sharing** (a decode row in a chunk step waits the chunk's whole time), and any eviction/residency story is worth ~10 % of one step. The residency half is not censusable: the arena / expert slab / heat engine / prefill stager have **no metric at all** — they are read from the boot ledger, `/stats`, and `/diagnostics`.
|
||
|
||
## Bucket, then lag-test, before believing a cause
|
||
|
||
Bucket `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` from the *same* scrape, step ≤ 30 s (a 15 s scrape against a 120 s step samples one in eight). Per-stream rate = decode gauge ÷ `vllm:num_requests_running`; the tps gauges are ~1 s decayed batch-pooled windows.
|
||
|
||
Then shift the prefill series by lags and re-bucket:
|
||
|
||
- lag-0-only ⇒ step-time sharing — the chunk is holding the device.
|
||
- a lagged tail (2–14 min) ⇒ something evicted (arena loan, expert slab, KV).
|
||
- bursts in `vllm:num_preemptions_total` ⇒ a preempted sequence stopped and is re-prefilling from token 0: a stall, not sludge.
|
||
|
||
Also bucket TTFT (`histogram_quantile` over `vllm:time_to_first_token_seconds_bucket`) by the same series: the decode gauge only sees sequences already running, so an arriving request pays the same steps and shows in no decode number.
|
||
|
||
## Metric traps
|
||
|
||
- **Latency is `vllm:`, cadence is `radiance:`.** TTFT / inter-token / per-output-token / e2e / decode-time histograms and the running/waiting gauges exist only under the vLLM compatibility prefix. Preemption is `vllm:num_preemptions_total` (counter); `radiance:num_requests_preempted` is the *instant* depth. A `radiance:`-only query finds no latency and no preemption and reads as "the endpoint never stalls anyone".
|
||
- **A 2-minute mean step time dilutes the chunk** — if ~5 % of steps carry a third of the step time, `step_seconds/steps` cannot move. Use the phase ratio.
|
||
- **`prefix_hit_cut_tokens_total` is prefill load you created yourself**: on a hybrid (linear + paged) model a hit reuses only to the last linear-state checkpoint, so cached tokens replay as prefill. A high prefill rate is not always a client sending new tokens.
|
||
- Radiance's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`, `token_queue_wait`) may be exported but unscraped; per-request forensics then mean `curl localhost:7770/stats` on the endpoint's host, and `/diagnostics` answers "is the *box* the limit" in one call (configuration drift, VRAM split, per-card link chain and clocks, findings with fixes).
|
||
- Rootless container logs: read as the deploy user (`podman logs <svc>`), never `sudo podman logs`; check where the serve script redirects before trusting a log file.
|
||
- In a temporary/remote shell, zsh aborts on a non-matching glob — quote or guard `/sys/...` wildcards.
|
||
|
||
## Knob semantics (radiance 1.8.x, GUIDE §4.5; pinned source is referee)
|
||
|
||
- `--prefill-interleave N` — the **decoder dial**: N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing. Costs the prefilling prompt's TTFT and a queued request its first step. Counts as `engine_steps_held_total`.
|
||
- `--busy-prefill-chunk` — a **prefill-speed dial**, not decoder protection: `auto` reserves 1+spec rows per decoding sequence and 1/8 of the budget per waiting request, and the source refuses to halve for decode rows alone. Never below the prefill stager's smallest pass (named on the boot line) — below it every routed layer reads its pooled experts synchronously.
|
||
- `--prefill-span N` — needs `--max-num-batched-tokens` = N intervals; re-funds a *lone* prefill only, so it cannot buy back held steps while decoders run, and its extra intervals land on the arena that the expert slab loses.
|
||
- `--checkpoint-policy` — retention only, three names (`geometric-backoff` | `tip-only` | `keep-all`); snapshot writes happen at every interval whatever the policy says (only `--prefill-span` skips them), so `keep-all` buys a bounded replay for pinned host RAM and no extra PCIe. Shortening `--checkpoint-interval` does NOT fix deep divergence: the geometric ladder's coarsest scale is the largest `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt while every cut multiplies the writes.
|
||
- `--checkpoint-slots` is a **total**, `auto` = policy `held()` + 1, per `--max-num-seqs`, within an eighth of host RAM; `--max-num-batched-tokens` is the blunt rung — it raises the arena the slab loses and its resonance on an expert-tiered payload can be violent.
|
||
|
||
## Verify an arm, don't assume it
|
||
|
||
Flags may be passed conditionally by the serve script, so prove the flag reached argv (`tr '\0' ' ' < /proc/$(pidof <engine>)/cmdline`), then the counter the knob moves (`engine_steps_held_total` must leave zero), then the bucket pair. A deploy is not evidence a knob engaged. Rollback for each of these is its env line, not a rebuild.
|