docs: the pause-process trigger is unproven — say so, everywhere

I had written 'memory pressure from the 116 GiB iGPU runs' as the cause of the
pause-process death. journalctl -k records no OOM kill for that window and the
kernel says nothing else about it, so the claim was a guess wearing a finding's
clothes — in AGENTS.md, the x4 note, and the log. All three now state the
trigger is not established, and the note's 'co-tenants' patience' bullet no
longer asserts that causality either. AGENTS.md also gains the real lesson:
journalctl -k is the check, and an unknown cause does not need a story.
This commit is contained in:
Frank
2026-09-16 19:34:40 -07:00
parent 7e11d247a8
commit 4d6511d8b3
3 changed files with 18 additions and 11 deletions
+4 -2
View File
@@ -567,8 +567,10 @@ linked runbooks.
stale token is not enough, and `podman system migrate` segfaults in
`teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
first). Observed trigger 2026-09-16: memory pressure from an iGPU run that
committed 116 of the host's 121 GiB.
first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the
trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel
had no other word on it). Treat the pause process as host state that can die
for its own reasons, and reach for the reboot when it does.
- **A container cannot reach a `Network=host` service at the host's LAN IP.**
netavark DNATs *published* ports only: from a bridge container
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
@@ -210,11 +210,13 @@ distinguishes a real split from a mislabelled sweep.
3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted
runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this
build.)
- **A killed rootless podman pause process takes the whole host's container
management down.** After the experiment the pause process was gone (plausible
cause: memory pressure from the 116 GiB iGPU runs), and every `podman` command
for the user failed with `invalid internal status … could not find any running
process`. `podman system migrate` — the remedy podman itself names — **panics**
- **A dead rootless podman pause process takes the whole host's container
management down.** After the experiment the pause process was gone and every
`podman` command for the user failed with `invalid internal status … could not
find any running process`. The trigger is **not** established: `journalctl -k`
holds no OOM kill for that window, so the tempting reading ("the 116 GiB runs
squeezed it out") is a guess and is recorded as one.
`podman system migrate` — the remedy podman itself names — **panics**
in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds
podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind
`10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears
@@ -226,8 +228,10 @@ distinguishes a real split from a mislabelled sweep.
same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and
guarded by a strengthened seam (`tests/hermes_webui.bats`).
- **The memory budget includes the co-tenants' patience.** The iGPU runs held
116 of 121 GiB; the pause-process death above is the visible cost of running a
model that big on a host that also serves.
116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log
proves damage came of it, but running a model that big on a host that also
serves ~28 containers is asking the question — do it in a window, not
alongside.
## Conclusion
+3 -2
View File
@@ -687,8 +687,9 @@ timestamp: 2026-09-14
their experts with them (1.62 GiB per layer), the 12 attention layers are spread
every 4th across the depth, and the 32.8 GiB PLE must stay on the main device —
so KV cannot be consolidated on the 3090.
- **Incident, same evening:** the rootless podman pause process died (memory
pressure from the 116 GiB iGPU runs is the plausible cause). Every `podman`
- **Incident, same evening:** the rootless podman pause process died — trigger
NOT established (no OOM kill in `journalctl -k` for that window, so "the
116 GiB runs squeezed it out" is a guess, not a finding). Every `podman`
command then failed (`invalid internal status … could not find any running
process`), every container start failed on aardvark-dns binding `10.89.x.1:53`
(the rootless netns holding the bridges was gone), and all 28 quadlets looped.