chore(agent): commit the machine tier and the doctrine text the checkout was already running

Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and
`merge --ff-only` refuses while tracked paths are dirty. Discard is not available here
(workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed.

- `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already
  byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new
  prose — the commit exists so the trunk merge can proceed.
- `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat
  owns (AGENTS.md § Live-sync tiers).

`ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the
only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the
inert all-zero AWS example id — and they appear inside the two skill pages that teach
that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged
gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal
dies with this landing. No `--no-verify` was used: the gate ran and its verdict is
recorded here.

Refs #61
This commit is contained in:
Frank committed 2026-10-21 01:44:51 -07:00
1 parent 004962ffce
commit 8af40c402b
79 files changed
+2771 -332

No files matched your search

@@ -0,0 +1,59 @@
---
name: radiance-knob-arm-live-verify
description: "Arm a radiance scheduler/cadence knob on a live endpoint and prove it engaged: per-knob /metrics meter, argv-vs-default truth, the three behaviour probes, no bench."
---
# Arming a radiance knob on a live endpoint (no bench)
Use for a one-flag change on a radiance endpoint (e.g. `PREFILL_INTERLEAVE=0→1`,
`BUSY_PREFILL_CHUNK`, `CHECKPOINT_POLICY`) under the "swap, don't instrument" law:
edit the trio → deploy the one service → prove the flag reached the engine, the
engine's own counter moved, and behaviour survived. Never a bench unless Frank orders one.
## Before you set a value
- **Unset means default, and the engine prints what it resolved.** Grep
`/proc/$(pidof radiance)/cmdline` (`sudo tr '\0' ' ' < …`) *and* the container ledger
(`podman logs <svc>` as the deploy user — rootless storage, `sudo podman` finds nothing).
The ledger line names the resolved value and whether a budget clamped it
(`--checkpoint-slots 20 [auto] … of a budget of 31.5 GiB, bound by None`).
- **Prove the flag exists in the pinned build** before promising it: `--help` from the
running container image, not from a vendored tree. Inventing a name (`floor`) cost a
doc rewrite; `config_parse` refuses unknown enums at boot.
- Some flags are conditional in the serve script (e.g. `--prefill-interleave` is passed
only when the env value is `> 0`), so a zero→nonzero arm adds argv; re-check argv after deploy.
## Meter map (`/metrics`, names carry the `radiance:` prefix)
| change | meter that must move |
|---|---|
| `--prefill-interleave` | `engine_steps_held_total` (read 0 for a 34.7 h boot while the knob sat at 0; 126 within 4 min of arming) |
| `--busy-prefill-chunk` | `engine_prefill_chunks_capped_total` |
| step-time contention generally | decode-only `engine_step_seconds_decode_total/engine_steps_decode_total` vs `(step_seconds_total − step_seconds_decode_total)/(steps_total − steps_decode_total)` |
| prefix retention | `prefix_hit_cut_tokens_total` (labels: `no_snapshot`), `vllm:num_preemptions_total` |
Trap: **latency lives under `vllm:`, cadence under `radiance:`.** A `radiance:`-only query
finds no TTFT/ITP and no preemption and reads as "nothing stalls". The arena / expert slab /
heat engine / prefill stager have **no metric at all** — that half is boot ledger + `/stats`.
## Post-deploy proof, in order
1. argv contains the flag (or the ledger shows the resolved value).
2. `/health` 200 (`in_flight`, `queue_cap` come back in the body).
3. Three probes on the endpoint itself: exact-reply chat (`temperature 0`, expect the
literal word — reasoning tokens are normal in `reasoning_content`); a `tools` call
expecting a real `tool_calls` object; a **solid-colour PNG via `image_url`** (never the
legacy top-level `images` field) with `max_tokens ≥ 512` or thinking eats the budget and
a thinking model reads as blind.
4. The knob's meter left zero-state. If it didn't move, the arm did nothing — say so.
5. Effect is judged later from real traffic buckets, not from a synthetic load.
## Deploy cycle gotchas that cost the most time
- `deploy.sh` must run from the dev checkout literally on `main`, tree clean including
untracked files; it pushes its own `main`. Never from an agent worktree or the target.
- Name the service (`--service <svc>`); a bare `deploy.sh <host>` is a fleet-wide drain.
- Health binds only after the full load (~100 s here, gate budget much larger) — 503 during
load is not a fault.
- Post-deploy for a recently-benched LLM service: `systemctl show <svc> -p DropInPaths`
must be empty before believing anything.
## Rollback
One env line back to the previous value + redeploy the same service. Write the meter and
the rollback into the endpoint's vault page (that page is the owner), never into a new page.