docs: the pause-process trigger is unproven — say so, everywhere
I had written 'memory pressure from the 116 GiB iGPU runs' as the cause of the pause-process death. journalctl -k records no OOM kill for that window and the kernel says nothing else about it, so the claim was a guess wearing a finding's clothes — in AGENTS.md, the x4 note, and the log. All three now state the trigger is not established, and the note's 'co-tenants' patience' bullet no longer asserts that causality either. AGENTS.md also gains the real lesson: journalctl -k is the check, and an unknown cause does not need a story.
This commit is contained in:
@@ -567,8 +567,10 @@ linked runbooks.
|
||||
stale token is not enough, and `podman system migrate` segfaults in
|
||||
`teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
|
||||
quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
|
||||
first). Observed trigger 2026-09-16: memory pressure from an iGPU run that
|
||||
committed 116 of the host's 121 GiB.
|
||||
first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the
|
||||
trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel
|
||||
had no other word on it). Treat the pause process as host state that can die
|
||||
for its own reasons, and reach for the reboot when it does.
|
||||
- **A container cannot reach a `Network=host` service at the host's LAN IP.**
|
||||
netavark DNATs *published* ports only: from a bridge container
|
||||
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
|
||||
|
||||
@@ -210,11 +210,13 @@ distinguishes a real split from a mislabelled sweep.
|
||||
3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted
|
||||
runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this
|
||||
build.)
|
||||
- **A killed rootless podman pause process takes the whole host's container
|
||||
management down.** After the experiment the pause process was gone (plausible
|
||||
cause: memory pressure from the 116 GiB iGPU runs), and every `podman` command
|
||||
for the user failed with `invalid internal status … could not find any running
|
||||
process`. `podman system migrate` — the remedy podman itself names — **panics**
|
||||
- **A dead rootless podman pause process takes the whole host's container
|
||||
management down.** After the experiment the pause process was gone and every
|
||||
`podman` command for the user failed with `invalid internal status … could not
|
||||
find any running process`. The trigger is **not** established: `journalctl -k`
|
||||
holds no OOM kill for that window, so the tempting reading ("the 116 GiB runs
|
||||
squeezed it out") is a guess and is recorded as one.
|
||||
`podman system migrate` — the remedy podman itself names — **panics**
|
||||
in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds
|
||||
podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind
|
||||
`10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears
|
||||
@@ -226,8 +228,10 @@ distinguishes a real split from a mislabelled sweep.
|
||||
same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and
|
||||
guarded by a strengthened seam (`tests/hermes_webui.bats`).
|
||||
- **The memory budget includes the co-tenants' patience.** The iGPU runs held
|
||||
116 of 121 GiB; the pause-process death above is the visible cost of running a
|
||||
model that big on a host that also serves.
|
||||
116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log
|
||||
proves damage came of it, but running a model that big on a host that also
|
||||
serves ~28 containers is asking the question — do it in a window, not
|
||||
alongside.
|
||||
|
||||
## Conclusion
|
||||
|
||||
|
||||
@@ -687,8 +687,9 @@ timestamp: 2026-09-14
|
||||
their experts with them (1.62 GiB per layer), the 12 attention layers are spread
|
||||
every 4th across the depth, and the 32.8 GiB PLE must stay on the main device —
|
||||
so KV cannot be consolidated on the 3090.
|
||||
- **Incident, same evening:** the rootless podman pause process died (memory
|
||||
pressure from the 116 GiB iGPU runs is the plausible cause). Every `podman`
|
||||
- **Incident, same evening:** the rootless podman pause process died — trigger
|
||||
NOT established (no OOM kill in `journalctl -k` for that window, so "the
|
||||
116 GiB runs squeezed it out" is a guess, not a finding). Every `podman`
|
||||
command then failed (`invalid internal status … could not find any running
|
||||
process`), every container start failed on aardvark-dns binding `10.89.x.1:53`
|
||||
(the rootless netns holding the bridges was gone), and all 28 quadlets looped.
|
||||
|
||||
Reference in New Issue
Block a user