Frank
8af40c402b
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
60 lines
3.8 KiB
Markdown
60 lines
3.8 KiB
Markdown
---
|
||
name: radiance-knob-arm-live-verify
|
||
description: "Arm a radiance scheduler/cadence knob on a live endpoint and prove it engaged: per-knob /metrics meter, argv-vs-default truth, the three behaviour probes, no bench."
|
||
---
|
||
|
||
# Arming a radiance knob on a live endpoint (no bench)
|
||
|
||
Use for a one-flag change on a radiance endpoint (e.g. `PREFILL_INTERLEAVE=0→1`,
|
||
`BUSY_PREFILL_CHUNK`, `CHECKPOINT_POLICY`) under the "swap, don't instrument" law:
|
||
edit the trio → deploy the one service → prove the flag reached the engine, the
|
||
engine's own counter moved, and behaviour survived. Never a bench unless Frank orders one.
|
||
|
||
## Before you set a value
|
||
- **Unset means default, and the engine prints what it resolved.** Grep
|
||
`/proc/$(pidof radiance)/cmdline` (`sudo tr '\0' ' ' < …`) *and* the container ledger
|
||
(`podman logs <svc>` as the deploy user — rootless storage, `sudo podman` finds nothing).
|
||
The ledger line names the resolved value and whether a budget clamped it
|
||
(`--checkpoint-slots 20 [auto] … of a budget of 31.5 GiB, bound by None`).
|
||
- **Prove the flag exists in the pinned build** before promising it: `--help` from the
|
||
running container image, not from a vendored tree. Inventing a name (`floor`) cost a
|
||
doc rewrite; `config_parse` refuses unknown enums at boot.
|
||
- Some flags are conditional in the serve script (e.g. `--prefill-interleave` is passed
|
||
only when the env value is `> 0`), so a zero→nonzero arm adds argv; re-check argv after deploy.
|
||
|
||
## Meter map (`/metrics`, names carry the `radiance:` prefix)
|
||
| change | meter that must move |
|
||
|---|---|
|
||
| `--prefill-interleave` | `engine_steps_held_total` (read 0 for a 34.7 h boot while the knob sat at 0; 126 within 4 min of arming) |
|
||
| `--busy-prefill-chunk` | `engine_prefill_chunks_capped_total` |
|
||
| step-time contention generally | decode-only `engine_step_seconds_decode_total/engine_steps_decode_total` vs `(step_seconds_total − step_seconds_decode_total)/(steps_total − steps_decode_total)` |
|
||
| prefix retention | `prefix_hit_cut_tokens_total` (labels: `no_snapshot`), `vllm:num_preemptions_total` |
|
||
|
||
Trap: **latency lives under `vllm:`, cadence under `radiance:`.** A `radiance:`-only query
|
||
finds no TTFT/ITP and no preemption and reads as "nothing stalls". The arena / expert slab /
|
||
heat engine / prefill stager have **no metric at all** — that half is boot ledger + `/stats`.
|
||
|
||
## Post-deploy proof, in order
|
||
1. argv contains the flag (or the ledger shows the resolved value).
|
||
2. `/health` 200 (`in_flight`, `queue_cap` come back in the body).
|
||
3. Three probes on the endpoint itself: exact-reply chat (`temperature 0`, expect the
|
||
literal word — reasoning tokens are normal in `reasoning_content`); a `tools` call
|
||
expecting a real `tool_calls` object; a **solid-colour PNG via `image_url`** (never the
|
||
legacy top-level `images` field) with `max_tokens ≥ 512` or thinking eats the budget and
|
||
a thinking model reads as blind.
|
||
4. The knob's meter left zero-state. If it didn't move, the arm did nothing — say so.
|
||
5. Effect is judged later from real traffic buckets, not from a synthetic load.
|
||
|
||
## Deploy cycle gotchas that cost the most time
|
||
- `deploy.sh` must run from the dev checkout literally on `main`, tree clean including
|
||
untracked files; it pushes its own `main`. Never from an agent worktree or the target.
|
||
- Name the service (`--service <svc>`); a bare `deploy.sh <host>` is a fleet-wide drain.
|
||
- Health binds only after the full load (~100 s here, gate budget much larger) — 503 during
|
||
load is not a fault.
|
||
- Post-deploy for a recently-benched LLM service: `systemctl show <svc> -p DropInPaths`
|
||
must be empty before believing anything.
|
||
|
||
## Rollback
|
||
One env line back to the previous value + redeploy the same service. Write the meter and
|
||
the rollback into the endpoint's vault page (that page is the owner), never into a new page.
|