docs: the pause-process trigger is unproven — say so, everywhere

I had written 'memory pressure from the 116 GiB iGPU runs' as the cause of the
pause-process death. journalctl -k records no OOM kill for that window and the
kernel says nothing else about it, so the claim was a guess wearing a finding's
clothes — in AGENTS.md, the x4 note, and the log. All three now state the
trigger is not established, and the note's 'co-tenants' patience' bullet no
longer asserts that causality either. AGENTS.md also gains the real lesson:
journalctl -k is the check, and an unknown cause does not need a story.
This commit is contained in:
Frank committed 2026-09-16 19:34:40 -07:00
1 parent 7e11d247a8
commit 4d6511d8b3
3 files changed
+18 -11

No files matched your search

+4 -2
View File
@@ -567,8 +567,10 @@ linked runbooks.
stale token is not enough, and `podman system migrate` segfaults in stale token is not enough, and `podman system migrate` segfaults in
`teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28 `teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
quadlets return on their own — check `Linger=yes` and `WantedBy=default.target` quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
first). Observed trigger 2026-09-16: memory pressure from an iGPU run that first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the
committed 116 of the host's 121 GiB. trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel
had no other word on it). Treat the pause process as host state that can die
for its own reasons, and reach for the reboot when it does.
- **A container cannot reach a `Network=host` service at the host's LAN IP.** - **A container cannot reach a `Network=host` service at the host's LAN IP.**
netavark DNATs *published* ports only: from a bridge container netavark DNATs *published* ports only: from a bridge container
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the `192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
@@ -210,11 +210,13 @@ distinguishes a real split from a mislabelled sweep.
3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted 3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted
runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this
build.) build.)
- **A killed rootless podman pause process takes the whole host's container - **A dead rootless podman pause process takes the whole host's container
management down.** After the experiment the pause process was gone (plausible management down.** After the experiment the pause process was gone and every
cause: memory pressure from the 116 GiB iGPU runs), and every `podman` command `podman` command for the user failed with `invalid internal status … could not
for the user failed with `invalid internal status … could not find any running find any running process`. The trigger is **not** established: `journalctl -k`
process`. `podman system migrate` — the remedy podman itself names — **panics** holds no OOM kill for that window, so the tempting reading ("the 116 GiB runs
squeezed it out") is a guess and is recorded as one.
`podman system migrate` — the remedy podman itself names — **panics**
in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds
podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind
`10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears `10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears
@@ -226,8 +228,10 @@ distinguishes a real split from a mislabelled sweep.
same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and
guarded by a strengthened seam (`tests/hermes_webui.bats`). guarded by a strengthened seam (`tests/hermes_webui.bats`).
- **The memory budget includes the co-tenants' patience.** The iGPU runs held - **The memory budget includes the co-tenants' patience.** The iGPU runs held
116 of 121 GiB; the pause-process death above is the visible cost of running a 116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log
model that big on a host that also serves. proves damage came of it, but running a model that big on a host that also
serves ~28 containers is asking the question — do it in a window, not
alongside.
## Conclusion ## Conclusion
+3 -2
View File
@@ -687,8 +687,9 @@ timestamp: 2026-09-14
their experts with them (1.62 GiB per layer), the 12 attention layers are spread their experts with them (1.62 GiB per layer), the 12 attention layers are spread
every 4th across the depth, and the 32.8 GiB PLE must stay on the main device — every 4th across the depth, and the 32.8 GiB PLE must stay on the main device —
so KV cannot be consolidated on the 3090. so KV cannot be consolidated on the 3090.
- **Incident, same evening:** the rootless podman pause process died (memory - **Incident, same evening:** the rootless podman pause process died — trigger
pressure from the 116 GiB iGPU runs is the plausible cause). Every `podman` NOT established (no OOM kill in `journalctl -k` for that window, so "the
116 GiB runs squeezed it out" is a guess, not a finding). Every `podman`
command then failed (`invalid internal status … could not find any running command then failed (`invalid internal status … could not find any running
process`), every container start failed on aardvark-dns binding `10.89.x.1:53` process`), every container start failed on aardvark-dns binding `10.89.x.1:53`
(the rootless netns holding the bridges was gone), and all 28 quadlets looped. (the rootless netns holding the bridges was gone), and all 28 quadlets looped.