docs: the pause-process trigger is unproven — say so, everywhere
I had written 'memory pressure from the 116 GiB iGPU runs' as the cause of the pause-process death. journalctl -k records no OOM kill for that window and the kernel says nothing else about it, so the claim was a guess wearing a finding's clothes — in AGENTS.md, the x4 note, and the log. All three now state the trigger is not established, and the note's 'co-tenants' patience' bullet no longer asserts that causality either. AGENTS.md also gains the real lesson: journalctl -k is the check, and an unknown cause does not need a story.
This commit is contained in:
1 parent
7e11d247a8
commit
4d6511d8b3
3 files changed
+18
-11
No files matched your search
@@ -567,8 +567,10 @@ linked runbooks.
|
|||||||
stale token is not enough, and `podman system migrate` segfaults in
|
stale token is not enough, and `podman system migrate` segfaults in
|
||||||
`teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
|
`teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
|
||||||
quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
|
quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
|
||||||
first). Observed trigger 2026-09-16: memory pressure from an iGPU run that
|
first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the
|
||||||
committed 116 of the host's 121 GiB.
|
trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel
|
||||||
|
had no other word on it). Treat the pause process as host state that can die
|
||||||
|
for its own reasons, and reach for the reboot when it does.
|
||||||
- **A container cannot reach a `Network=host` service at the host's LAN IP.**
|
- **A container cannot reach a `Network=host` service at the host's LAN IP.**
|
||||||
netavark DNATs *published* ports only: from a bridge container
|
netavark DNATs *published* ports only: from a bridge container
|
||||||
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
|
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
|
||||||
|
|||||||
@@ -210,11 +210,13 @@ distinguishes a real split from a mislabelled sweep.
|
|||||||
3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted
|
3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted
|
||||||
runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this
|
runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this
|
||||||
build.)
|
build.)
|
||||||
- **A killed rootless podman pause process takes the whole host's container
|
- **A dead rootless podman pause process takes the whole host's container
|
||||||
management down.** After the experiment the pause process was gone (plausible
|
management down.** After the experiment the pause process was gone and every
|
||||||
cause: memory pressure from the 116 GiB iGPU runs), and every `podman` command
|
`podman` command for the user failed with `invalid internal status … could not
|
||||||
for the user failed with `invalid internal status … could not find any running
|
find any running process`. The trigger is **not** established: `journalctl -k`
|
||||||
process`. `podman system migrate` — the remedy podman itself names — **panics**
|
holds no OOM kill for that window, so the tempting reading ("the 116 GiB runs
|
||||||
|
squeezed it out") is a guess and is recorded as one.
|
||||||
|
`podman system migrate` — the remedy podman itself names — **panics**
|
||||||
in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds
|
in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds
|
||||||
podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind
|
podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind
|
||||||
`10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears
|
`10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears
|
||||||
@@ -226,8 +228,10 @@ distinguishes a real split from a mislabelled sweep.
|
|||||||
same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and
|
same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and
|
||||||
guarded by a strengthened seam (`tests/hermes_webui.bats`).
|
guarded by a strengthened seam (`tests/hermes_webui.bats`).
|
||||||
- **The memory budget includes the co-tenants' patience.** The iGPU runs held
|
- **The memory budget includes the co-tenants' patience.** The iGPU runs held
|
||||||
116 of 121 GiB; the pause-process death above is the visible cost of running a
|
116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log
|
||||||
model that big on a host that also serves.
|
proves damage came of it, but running a model that big on a host that also
|
||||||
|
serves ~28 containers is asking the question — do it in a window, not
|
||||||
|
alongside.
|
||||||
|
|
||||||
## Conclusion
|
## Conclusion
|
||||||
|
|
||||||
|
|||||||
@@ -687,8 +687,9 @@ timestamp: 2026-09-14
|
|||||||
their experts with them (1.62 GiB per layer), the 12 attention layers are spread
|
their experts with them (1.62 GiB per layer), the 12 attention layers are spread
|
||||||
every 4th across the depth, and the 32.8 GiB PLE must stay on the main device —
|
every 4th across the depth, and the 32.8 GiB PLE must stay on the main device —
|
||||||
so KV cannot be consolidated on the 3090.
|
so KV cannot be consolidated on the 3090.
|
||||||
- **Incident, same evening:** the rootless podman pause process died (memory
|
- **Incident, same evening:** the rootless podman pause process died — trigger
|
||||||
pressure from the 116 GiB iGPU runs is the plausible cause). Every `podman`
|
NOT established (no OOM kill in `journalctl -k` for that window, so "the
|
||||||
|
116 GiB runs squeezed it out" is a guess, not a finding). Every `podman`
|
||||||
command then failed (`invalid internal status … could not find any running
|
command then failed (`invalid internal status … could not find any running
|
||||||
process`), every container start failed on aardvark-dns binding `10.89.x.1:53`
|
process`), every container start failed on aardvark-dns binding `10.89.x.1:53`
|
||||||
(the rootless netns holding the bridges was gone), and all 28 quadlets looped.
|
(the rootless netns holding the bridges was gone), and all 28 quadlets looped.
|
||||||
|
|||||||
Reference in new issue
Block a user