docs(frank-archon): record the crash signature — clean-looking exit, silent engine
Investigation of the 2026-11-04 13:09 crash, recorded as one Law on the endpoint
page because the fault is not the durable part — the blind spot is.
What the crash looked like from outside
Three concurrent triple-digit-thousand-token agent turns (89.5k / 61k / 109.5k
prompt tokens, tools=14) prefilled simultaneously from three omp sessions on
blankpad. At 13:09:40 the engine failed all three in flight with its own text — the engine stopped on an error and is not serving (invalid argument) — then tore
down silently and exited 1 at 13:09:53. Restart=always reloaded it in 65 s; /health 200, up=1 from 13:11:30, and a real generation (pong, 36 tokens)
confirms the loop serves.
The two traps it exposed (now in the page's Laws)
ExecMainStatus=0 on a crash. The unit's ExecStart ends in exec podman wait, whose own status is 0; the container's code is the bare digit podman wait prints on stdout. Result=success + NRestarts=1 is the normal
fingerprint of a crash — reading unit properties instead of the journal lines
says "clean exit" and someone re-checks the wrong thing.
The engine logs nothing on its fatal path. No stderr line, no message, no
core: the only copy of the reason is the client-facing error text. Diagnose
from ~/.omp/logs/omp.<date>.<pid>.log (agent turn ended with provider error) or the OpenWebUI toast, never from an empty journal.
Ruled out with evidence: no OOM (kernel log clean), no GPU hang/reset, no MCE, no
swap pressure, no client-disconnect storm (client_disconnect_total constant).
The kernel logged only KFD svm_range_deferred_list_work churn in the teardown
second — twice all boot, both in that second, i.e. teardown, not cause. The
engine was already straining: prefill_tokens_per_second collapsed
(805 → 1.7 t/s) in its last scrape while prefix_cache_evictions_total climbed.
The triggering payload shape is recorded qualitatively (concurrent long prefills)
— the rates stay out of the page, per the docs law.
# docs(frank-archon): record the crash signature — clean-looking exit, silent engine
Investigation of the 2026-11-04 13:09 crash, recorded as one Law on the endpoint
page because the *fault* is not the durable part — the *blind spot* is.
## What the crash looked like from outside
Three concurrent triple-digit-thousand-token agent turns (89.5k / 61k / 109.5k
prompt tokens, `tools=14`) prefilled simultaneously from three omp sessions on
blankpad. At 13:09:40 the engine failed all three in flight with its own text —
`the engine stopped on an error and is not serving (invalid argument)` — then tore
down silently and exited **1** at 13:09:53. `Restart=always` reloaded it in 65 s;
`/health` 200, `up=1` from 13:11:30, and a real generation (`pong`, 36 tokens)
confirms the loop serves.
## The two traps it exposed (now in the page's Laws)
1. **`ExecMainStatus=0` on a crash.** The unit's `ExecStart` ends in `exec podman
wait`, whose own status is 0; the container's code is the bare digit `podman
wait` prints on stdout. `Result=success` + `NRestarts=1` is the *normal*
fingerprint of a crash — reading unit properties instead of the journal lines
says "clean exit" and someone re-checks the wrong thing.
2. **The engine logs nothing on its fatal path.** No stderr line, no message, no
core: the only copy of the reason is the client-facing error text. Diagnose
from `~/.omp/logs/omp.<date>.<pid>.log` (`agent turn ended with provider
error`) or the OpenWebUI toast, never from an empty journal.
Ruled out with evidence: no OOM (kernel log clean), no GPU hang/reset, no MCE, no
swap pressure, no client-disconnect storm (`client_disconnect_total` constant).
The kernel logged only KFD `svm_range_deferred_list_work` churn in the teardown
second — twice all boot, both in that second, i.e. teardown, not cause. The
engine was already straining: `prefill_tokens_per_second` collapsed
(805 → 1.7 t/s) in its last scrape while `prefix_cache_evictions_total` climbed.
The triggering payload shape is recorded qualitatively (concurrent long prefills)
— the rates stay out of the page, per the docs law.
---
cwd: /home/blank/.omp/wt/wt-20261104-104444-6d1fc1f · session: 01a1a7eb-c584-72d3-9d7e-c02d981cd662
A 2026-11-04 crash left no trace in the engine's own log and reported
Result=success with NRestarts=1: the unit's ExecStart ends in 'exec podman
wait', whose exit status is 0, while the container's real code is printed to
stdout. The only copy of the reason was the client-facing error text. Both
traps now live in the endpoint page's Laws, with the observed trigger shape
(concurrent triple-digit-thousand-token prefills) and the instruments that
ruled out OOM, GPU hang, MCE and swap pressure.
dank
merged commit 9ff7ce6a12 into main2026-11-04 13:19:50 -08:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
docs(frank-archon): record the crash signature — clean-looking exit, silent engine
Investigation of the 2026-11-04 13:09 crash, recorded as one Law on the endpoint
page because the fault is not the durable part — the blind spot is.
What the crash looked like from outside
Three concurrent triple-digit-thousand-token agent turns (89.5k / 61k / 109.5k
prompt tokens,
tools=14) prefilled simultaneously from three omp sessions onblankpad. At 13:09:40 the engine failed all three in flight with its own text —
the engine stopped on an error and is not serving (invalid argument)— then toredown silently and exited 1 at 13:09:53.
Restart=alwaysreloaded it in 65 s;/health200,up=1from 13:11:30, and a real generation (pong, 36 tokens)confirms the loop serves.
The two traps it exposed (now in the page's Laws)
ExecMainStatus=0on a crash. The unit'sExecStartends inexec podman wait, whose own status is 0; the container's code is the bare digitpodman waitprints on stdout.Result=success+NRestarts=1is the normalfingerprint of a crash — reading unit properties instead of the journal lines
says "clean exit" and someone re-checks the wrong thing.
core: the only copy of the reason is the client-facing error text. Diagnose
from
~/.omp/logs/omp.<date>.<pid>.log(agent turn ended with provider error) or the OpenWebUI toast, never from an empty journal.Ruled out with evidence: no OOM (kernel log clean), no GPU hang/reset, no MCE, no
swap pressure, no client-disconnect storm (
client_disconnect_totalconstant).The kernel logged only KFD
svm_range_deferred_list_workchurn in the teardownsecond — twice all boot, both in that second, i.e. teardown, not cause. The
engine was already straining:
prefill_tokens_per_secondcollapsed(805 → 1.7 t/s) in its last scrape while
prefix_cache_evictions_totalclimbed.The triggering payload shape is recorded qualitatively (concurrent long prefills)
— the rates stay out of the page, per the docs law.
cwd: /home/blank/.omp/wt/wt-20261104-104444-6d1fc1f · session: 01a1a7eb-c584-72d3-9d7e-c02d981cd662