33 lines
4.1 KiB
Markdown
33 lines
4.1 KiB
Markdown
---
|
||
name: radiance-endpoint-silent-wedge-triage
|
||
description: "Diagnose a radiance (frank-archon) endpoint that is wedged or crawling while systemd says active(running) and /health answers 200 — separate engine-state wedge from config, hardware, swap and clock causes, and prove the restart fixed it."
|
||
---
|
||
|
||
# Silent wedge: `active (running)` + `/health` 200, but no throughput
|
||
|
||
For radiance endpoints (`frank-archon`, root unit whose `ExecStart` is `exec podman wait`). The unit can only exit when the container dies, so a stalled engine loop **never** triggers `Restart=always`, `up` stays `1`, and Alertmanager stays empty (every rule keys on `up==0`, `num_requests_waiting>0`, or TTFT p95). Verified 2026-10-30 21:00 on dankserver.
|
||
|
||
## Triage ladder (in this order — each step kills a class)
|
||
|
||
1. **Front end vs engine split.** `curl -w '%{time_total}' /health` vs `/v1/models`. A 0.4 ms `/health` beside a 20 s `/v1/models` timeout = front end alive, engine loop stalled.
|
||
2. **Unit truth, not the port.** `systemctl status frank-archon -n 0` + `systemctl show -p MainPID,DropInPaths,ExecMainStartTimestamp`. A 4 h-old unit on a 2-week-old host means a restart happened today — cross-check `journalctl -u <svc> --since <day>` for `Starting/Stopping/exit` and `git log -- scripts/<svc>.*` to see whether a deploy caused it. Non-empty `DropInPaths` = leftover bench arm (separate trap).
|
||
3. **Container storage namespace.** The unit is `User=dank`, so the container lives in **rootless** storage: `sudo podman ps -a` lists nothing and `podman inspect` says `no such object` — query as the user, without sudo.
|
||
4. **Engine's own voice (the decisive artifact).** `podman logs --timestamps frank-archon | grep "E device:"`. `a synchronisation has seen no queue advance for 5 s` naming `barrier-and dep[0]=1` on both ranks = the TP all-reduce never completed. Timestamp the first occurrence; that is the incident start.
|
||
5. **Rule out the engine's own suggested causes before believing anything else:** `cat /sys/module/amdgpu/parameters/cwsr_enable` (its message blames `cwsr_enable=0`; `1` falsifies it). Swap: read `/proc/<engine-pid>/status` VmSwap — do NOT infer from `free`/`/proc/swaps` totals (host-wide swap can be 32 GiB while the engine holds 378 MiB). Clocks during load: fire a request, then `rocm-smi --showclocks --showperflevel --showpower` + `/sys/class/drm/cardN/device/gpu_busy_percent` — 100 % busy at full sclk means neither throttle nor starvation.
|
||
6. **Quantify the collapse from counters, not vibes.** `radiance:engine_step_seconds_decode_total` ÷ `radiance:engine_steps_decode_total` = mean decode step time; diff two scrapes (Δseconds ÷ Δsteps) for the current rate. Compare to the shape's recorded baseline in the endpoint's vault page (15.5 ms at frank-archon's 4×200000 fp8 shape; 209 ms = 13× collapse).
|
||
|
||
## Proof of fix (same-request A/B, not a health check)
|
||
|
||
Submit one fixed request before and after the restart and diff the engine's own `timings` object — an 8-token probe measures only first-token noise, use `max_tokens>=128`:
|
||
|
||
```
|
||
curl -sS -m 240 -o /tmp/q.json -X POST :7770/v1/chat/completions \
|
||
-H 'Content-Type: application/json' -d @/tmp/query.json # then read timings.predicted_per_token_ms / predicted_per_second / ttft_ms / draft_n / draft_n_accepted
|
||
```
|
||
|
||
Before/after on the identical request: 90.76 → 11.06 ms/token, 11.02 → 90.40 t/s, ttft 420 → 148 ms, acceptance unchanged (266/266). Then re-read the boot ledger (`elastic / kv cache / vram_weights / unclaimed / sampler rows`) and confirm it matches the page's recorded ledger — a successful restart proves nothing about the flags. Close with the restore oracle: `is-active` **and** `is-enabled` **and** no `/run/systemd/system/<unit>` mask symlink **and** container owned by the unit **and** `DropInPaths` empty.
|
||
|
||
## Law
|
||
|
||
A wedged radiance engine has **no in-place recovery** (its own wording: "a queue the driver evicted and could not restore") — `systemctl restart` is the only cure, and the fix must be proven by a rate, never by `/health`. Durable gap: no alert fires on a live-but-stalled endpoint; the missing rule is a decode-step-time ratio gated on `up==1` in `shared/prometheus/rules/llm-fleet.yml`.
|