command-center/agent/managed-skills/radiance-knob-arm-live-verify/SKILL.md

name, description

name	description
radiance-knob-arm-live-verify	Arm a radiance scheduler/cadence knob on a live endpoint and prove it engaged: per-knob /metrics meter, argv-vs-default truth, the three behaviour probes, no bench.

Arming a radiance knob on a live endpoint (no bench)

Use for a one-flag change on a radiance endpoint (e.g. PREFILL_INTERLEAVE=0→1, BUSY_PREFILL_CHUNK, CHECKPOINT_POLICY) under the "swap, don't instrument" law: edit the trio → deploy the one service → prove the flag reached the engine, the engine's own counter moved, and behaviour survived. Never a bench unless Frank orders one.

Before you set a value

• Unset means default, and the engine prints what it resolved. Grep /proc/$(pidof radiance)/cmdline (sudo tr '\0' ' ' < …) and the container ledger (podman logs <svc> as the deploy user — rootless storage, sudo podman finds nothing). The ledger line names the resolved value and whether a budget clamped it (--checkpoint-slots 20 [auto] … of a budget of 31.5 GiB, bound by None).
• Prove the flag exists in the pinned build before promising it: --help from the running container image, not from a vendored tree. Inventing a name (floor) cost a doc rewrite; config_parse refuses unknown enums at boot.
• Some flags are conditional in the serve script (e.g. --prefill-interleave is passed only when the env value is > 0), so a zero→nonzero arm adds argv; re-check argv after deploy.

Meter map (/metrics, names carry the radiance: prefix)

change	meter that must move
--prefill-interleave	engine_steps_held_total (read 0 for a 34.7 h boot while the knob sat at 0; 126 within 4 min of arming)
--busy-prefill-chunk	engine_prefill_chunks_capped_total
step-time contention generally	decode-only engine_step_seconds_decode_total/engine_steps_decode_total vs (step_seconds_total − step_seconds_decode_total)/(steps_total − steps_decode_total)
prefix retention	prefix_hit_cut_tokens_total (labels: no_snapshot), vllm:num_preemptions_total

Trap: latency lives under vllm:, cadence under radiance:. A radiance:-only query finds no TTFT/ITP and no preemption and reads as "nothing stalls". The arena / expert slab / heat engine / prefill stager have no metric at all — that half is boot ledger + /stats.

Post-deploy proof, in order

1. argv contains the flag (or the ledger shows the resolved value).
2. /health 200 (in_flight, queue_cap come back in the body).
3. Three probes on the endpoint itself: exact-reply chat (temperature 0, expect the literal word — reasoning tokens are normal in reasoning_content); a tools call expecting a real tool_calls object; a solid-colour PNG via image_url (never the legacy top-level images field) with max_tokens ≥ 512 or thinking eats the budget and a thinking model reads as blind.
4. The knob's meter left zero-state. If it didn't move, the arm did nothing — say so.
5. Effect is judged later from real traffic buckets, not from a synthetic load.

Deploy cycle gotchas that cost the most time

• deploy.sh must run from the dev checkout literally on main, tree clean including untracked files; it pushes its own main. Never from an agent worktree or the target.
• Name the service (--service <svc>); a bare deploy.sh <host> is a fleet-wide drain.
• Health binds only after the full load (~100 s here, gate budget much larger) — 503 during load is not a fault.
• Post-deploy for a recently-benched LLM service: systemctl show <svc> -p DropInPaths must be empty before believing anything.

Rollback

One env line back to the previous value + redeploy the same service. Write the meter and the rollback into the endpoint's vault page (that page is the owner), never into a new page.