docs(frank-archon): record the crash signature — clean-looking exit, silent engine

A 2026-11-04 crash left no trace in the engine's own log and reported
Result=success with NRestarts=1: the unit's ExecStart ends in 'exec podman
wait', whose exit status is 0, while the container's real code is printed to
stdout. The only copy of the reason was the client-facing error text. Both
traps now live in the endpoint page's Laws, with the observed trigger shape
(concurrent triple-digit-thousand-token prefills) and the instruments that
ruled out OOM, GPU hang, MCE and swap pressure.
This commit is contained in:
Frank committed 2026-11-04 13:18:43 -08:00
1 parent fee8121d47
commit f25be876fb
1 file changed
+15
+15
View File
@@ -525,6 +525,21 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
- `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
the disk tier silently never opens. The tier does survive a payload swap when arch and
tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
- **A radiance crash reads as a clean exit unless you know where to look, and the engine never
says why.** The unit's `ExecStart` ends in `exec podman wait`, so `systemctl show -p
ExecMainStatus` reports **0** after a container death — that is `podman wait`'s own status, not
the engine's — and `Result=success` beside `NRestarts=1` is the *normal* fingerprint of a
crash. The container's real code is the bare digit `podman wait` prints to stdout
(`serve-frank-archon.sh[NNNN]: 1`). The engine's fatal path writes **nothing** to its log: no
stderr line, no message before the exit and no core. The only copy of the reason is the error
the client received — `the engine stopped on an error and is not serving (<errno>)`, where the
parenthesis is the failing call's errno text (`invalid argument` = EINVAL, 2026-11-04). So
grep the *client* logs before concluding the journal is empty: omp keeps the provider error in
`~/.omp/logs/omp.<date>.<pid>.log` (`agent turn ended with provider error`), and OpenWebUI
shows it as the chat's error toast. Seen once (2026-11-04, restart counter 1 after 3d20h up):
three concurrent triple-digit-thousand-token agent turns prefilled at once, prefill rate
collapsed, all in-flight streams failed at the same instant, then a silent teardown — kernel
logged only the KFD SVM workqueue churning, with no OOM, GPU hang, MCE or swap pressure.
### Dead ends — do not re-try