---
name: radiance-knob-arm-live-verify
description: "Arm a radiance scheduler/cadence knob on a live endpoint and prove it engaged: per-knob /metrics meter, argv-vs-default truth, the three behaviour probes, no bench."
---

# Arming a radiance knob on a live endpoint (no bench)

Use for a one-flag change on a radiance endpoint (e.g. `PREFILL_INTERLEAVE=0→1`,
`BUSY_PREFILL_CHUNK`, `CHECKPOINT_POLICY`) under the "swap, don't instrument" law:
edit the trio → deploy the one service → prove the flag reached the engine, the
engine's own counter moved, and behaviour survived. Never a bench unless Frank orders one.

## Before you set a value
- **Unset means default, and the engine prints what it resolved.** Grep
  `/proc/$(pidof radiance)/cmdline` (`sudo tr '\0' ' ' < …`) *and* the container ledger
  (`podman logs <svc>` as the deploy user — rootless storage, `sudo podman` finds nothing).
  The ledger line names the resolved value and whether a budget clamped it
  (`--checkpoint-slots 20 [auto] … of a budget of 31.5 GiB, bound by None`).
- **Prove the flag exists in the pinned build** before promising it: `--help` from the
  running container image, not from a vendored tree. Inventing a name (`floor`) cost a
  doc rewrite; `config_parse` refuses unknown enums at boot.
- Some flags are conditional in the serve script (e.g. `--prefill-interleave` is passed
  only when the env value is `> 0`), so a zero→nonzero arm adds argv; re-check argv after deploy.

## Meter map (`/metrics`, names carry the `radiance:` prefix)
| change | meter that must move |
|---|---|
| `--prefill-interleave` | `engine_steps_held_total` (read 0 for a 34.7 h boot while the knob sat at 0; 126 within 4 min of arming) |
| `--busy-prefill-chunk` | `engine_prefill_chunks_capped_total` |
| step-time contention generally | decode-only `engine_step_seconds_decode_total/engine_steps_decode_total` vs `(step_seconds_total − step_seconds_decode_total)/(steps_total − steps_decode_total)` |
| prefix retention | `prefix_hit_cut_tokens_total` (labels: `no_snapshot`), `vllm:num_preemptions_total` |

Trap: **latency lives under `vllm:`, cadence under `radiance:`.** A `radiance:`-only query
finds no TTFT/ITP and no preemption and reads as "nothing stalls". The arena / expert slab /
heat engine / prefill stager have **no metric at all** — that half is boot ledger + `/stats`.

## Post-deploy proof, in order
1. argv contains the flag (or the ledger shows the resolved value).
2. `/health` 200 (`in_flight`, `queue_cap` come back in the body).
3. Three probes on the endpoint itself: exact-reply chat (`temperature 0`, expect the
   literal word — reasoning tokens are normal in `reasoning_content`); a `tools` call
   expecting a real `tool_calls` object; a **solid-colour PNG via `image_url`** (never the
   legacy top-level `images` field) with `max_tokens ≥ 512` or thinking eats the budget and
   a thinking model reads as blind.
4. The knob's meter left zero-state. If it didn't move, the arm did nothing — say so.
5. Effect is judged later from real traffic buckets, not from a synthetic load.

## Deploy cycle gotchas that cost the most time
- `deploy.sh` must run from the dev checkout literally on `main`, tree clean including
  untracked files; it pushes its own `main`. Never from an agent worktree or the target.
- Name the service (`--service <svc>`); a bare `deploy.sh <host>` is a fleet-wide drain.
- Health binds only after the full load (~100 s here, gate budget much larger) — 503 during
  load is not a fault.
- Post-deploy for a recently-benched LLM service: `systemctl show <svc> -p DropInPaths`
  must be empty before believing anything.

## Rollback
One env line back to the previous value + redeploy the same service. Write the meter and
the rollback into the endpoint's vault page (that page is the owner), never into a new page.