command-center/agent/managed-skills/radiance-endpoint-silent-wedge-triage/SKILL.md

name, description

name	description
radiance-endpoint-silent-wedge-triage	Diagnose a radiance (frank-archon) endpoint that is wedged or crawling while systemd says active(running) and /health answers 200 — separate engine-state wedge from config, hardware, swap and clock causes, and prove the restart fixed it.

Silent wedge: active (running) + /health 200, but no throughput

For radiance endpoints (frank-archon, root unit whose ExecStart is exec podman wait). The unit can only exit when the container dies, so a stalled engine loop never triggers Restart=always, up stays 1, and Alertmanager stays empty (every rule keys on up==0, num_requests_waiting>0, or TTFT p95). Verified 2026-10-30 21:00 on dankserver.

Triage ladder (in this order — each step kills a class)

1. Front end vs engine split. curl -w '%{time_total}' /health vs /v1/models. A 0.4 ms /health beside a 20 s /v1/models timeout = front end alive, engine loop stalled.
2. Unit truth, not the port. systemctl status frank-archon -n 0 + systemctl show -p MainPID,DropInPaths,ExecMainStartTimestamp. A 4 h-old unit on a 2-week-old host means a restart happened today — cross-check journalctl -u <svc> --since <day> for Starting/Stopping/exit and git log -- scripts/<svc>.* to see whether a deploy caused it. Non-empty DropInPaths = leftover bench arm (separate trap).
3. Container storage namespace. The unit is User=dank, so the container lives in rootless storage: sudo podman ps -a lists nothing and podman inspect says no such object — query as the user, without sudo.
4. Engine's own voice (the decisive artifact). podman logs --timestamps frank-archon | grep "E device:". a synchronisation has seen no queue advance for 5 s naming barrier-and dep[0]=1 on both ranks = the TP all-reduce never completed. Timestamp the first occurrence; that is the incident start.
5. Rule out the engine's own suggested causes before believing anything else: cat /sys/module/amdgpu/parameters/cwsr_enable (its message blames cwsr_enable=0; 1 falsifies it). Swap: read /proc/<engine-pid>/status VmSwap — do NOT infer from free//proc/swaps totals (host-wide swap can be 32 GiB while the engine holds 378 MiB). Clocks during load: fire a request, then rocm-smi --showclocks --showperflevel --showpower + /sys/class/drm/cardN/device/gpu_busy_percent — 100 % busy at full sclk means neither throttle nor starvation.
6. Quantify the collapse from counters, not vibes. radiance:engine_step_seconds_decode_total ÷ radiance:engine_steps_decode_total = mean decode step time; diff two scrapes (Δseconds ÷ Δsteps) for the current rate. Compare to the shape's recorded baseline in the endpoint's vault page (15.5 ms at frank-archon's 4×200000 fp8 shape; 209 ms = 13× collapse).

Proof of fix (same-request A/B, not a health check)

Submit one fixed request before and after the restart and diff the engine's own timings object — an 8-token probe measures only first-token noise, use max_tokens>=128:

curl -sS -m 240 -o /tmp/q.json -X POST :7770/v1/chat/completions \
  -H 'Content-Type: application/json' -d @/tmp/query.json   # then read timings.predicted_per_token_ms / predicted_per_second / ttft_ms / draft_n / draft_n_accepted

Before/after on the identical request: 90.76 → 11.06 ms/token, 11.02 → 90.40 t/s, ttft 420 → 148 ms, acceptance unchanged (266/266). Then re-read the boot ledger (elastic / kv cache / vram_weights / unclaimed / sampler rows) and confirm it matches the page's recorded ledger — a successful restart proves nothing about the flags. Close with the restore oracle: is-active and is-enabled and no /run/systemd/system/<unit> mask symlink and container owned by the unit and DropInPaths empty.

Law

A wedged radiance engine has no in-place recovery (its own wording: "a queue the driver evicted and could not restore") — systemctl restart is the only cure, and the fix must be proven by a rate, never by /health. Durable gap: no alert fires on a live-but-stalled endpoint; the missing rule is a decode-step-time ratio gated on up==1 in shared/prometheus/rules/llm-fleet.yml.