From f25be876fb6ec7260d7753920689790812bd21df Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 4 Nov 2026 13:18:43 -0800 Subject: [PATCH] =?UTF-8?q?docs(frank-archon):=20record=20the=20crash=20si?= =?UTF-8?q?gnature=20=E2=80=94=20clean-looking=20exit,=20silent=20engine?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A 2026-11-04 crash left no trace in the engine's own log and reported Result=success with NRestarts=1: the unit's ExecStart ends in 'exec podman wait', whose exit status is 0, while the container's real code is printed to stdout. The only copy of the reason was the client-facing error text. Both traps now live in the endpoint page's Laws, with the observed trigger shape (concurrent triple-digit-thousand-token prefills) and the instruments that ruled out OOM, GPU hang, MCE and swap pressure. --- serverAppConfigs-docs/LLM/frank-archon.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index 6acb70c..dd1a11d 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -525,6 +525,21 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). - `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or the disk tier silently never opens. The tier does survive a payload swap when arch and tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit). +- **A radiance crash reads as a clean exit unless you know where to look, and the engine never + says why.** The unit's `ExecStart` ends in `exec podman wait`, so `systemctl show -p + ExecMainStatus` reports **0** after a container death — that is `podman wait`'s own status, not + the engine's — and `Result=success` beside `NRestarts=1` is the *normal* fingerprint of a + crash. The container's real code is the bare digit `podman wait` prints to stdout + (`serve-frank-archon.sh[NNNN]: 1`). The engine's fatal path writes **nothing** to its log: no + stderr line, no message before the exit and no core. The only copy of the reason is the error + the client received — `the engine stopped on an error and is not serving ()`, where the + parenthesis is the failing call's errno text (`invalid argument` = EINVAL, 2026-11-04). So + grep the *client* logs before concluding the journal is empty: omp keeps the provider error in + `~/.omp/logs/omp...log` (`agent turn ended with provider error`), and OpenWebUI + shows it as the chat's error toast. Seen once (2026-11-04, restart counter 1 after 3d20h up): + three concurrent triple-digit-thousand-token agent turns prefilled at once, prefill rate + collapsed, all in-flight streams failed at the same instant, then a silent teardown — kernel + logged only the KFD SVM workqueue churning, with no OOM, GPU hang, MCE or swap pressure. ### Dead ends — do not re-try -- 2.54.0