diff --git a/AGENTS.md b/AGENTS.md index 1afa439..9914e43 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -567,8 +567,10 @@ linked runbooks. stale token is not enough, and `podman system migrate` segfaults in `teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28 quadlets return on their own — check `Linger=yes` and `WantedBy=default.target` - first). Observed trigger 2026-09-16: memory pressure from an iGPU run that - committed 116 of the host's 121 GiB. + first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the + trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel + had no other word on it). Treat the pause process as host state that can die + for its own reasons, and reach for the reboot when it does. - **A container cannot reach a `Network=host` service at the host's LAN IP.** netavark DNATs *published* ports only: from a bridge container `192.168.0.143:` answers while `192.168.0.143:` and the diff --git a/serverAppConfigs-docs/LLM/x4-pair-experiment.md b/serverAppConfigs-docs/LLM/x4-pair-experiment.md index beeb553..f83acef 100644 --- a/serverAppConfigs-docs/LLM/x4-pair-experiment.md +++ b/serverAppConfigs-docs/LLM/x4-pair-experiment.md @@ -210,11 +210,13 @@ distinguishes a real split from a mislabelled sweep. 3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this build.) -- **A killed rootless podman pause process takes the whole host's container - management down.** After the experiment the pause process was gone (plausible - cause: memory pressure from the 116 GiB iGPU runs), and every `podman` command - for the user failed with `invalid internal status … could not find any running - process`. `podman system migrate` — the remedy podman itself names — **panics** +- **A dead rootless podman pause process takes the whole host's container + management down.** After the experiment the pause process was gone and every + `podman` command for the user failed with `invalid internal status … could not + find any running process`. The trigger is **not** established: `journalctl -k` + holds no OOM kill for that window, so the tempting reading ("the 116 GiB runs + squeezed it out") is a guess and is recorded as one. + `podman system migrate` — the remedy podman itself names — **panics** in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind `10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears @@ -226,8 +228,10 @@ distinguishes a real split from a mislabelled sweep. same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and guarded by a strengthened seam (`tests/hermes_webui.bats`). - **The memory budget includes the co-tenants' patience.** The iGPU runs held - 116 of 121 GiB; the pause-process death above is the visible cost of running a - model that big on a host that also serves. + 116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log + proves damage came of it, but running a model that big on a host that also + serves ~28 containers is asking the question — do it in a window, not + alongside. ## Conclusion diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 86a2449..4d776c1 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -687,8 +687,9 @@ timestamp: 2026-09-14 their experts with them (1.62 GiB per layer), the 12 attention layers are spread every 4th across the depth, and the 32.8 GiB PLE must stay on the main device — so KV cannot be consolidated on the 3090. -- **Incident, same evening:** the rootless podman pause process died (memory - pressure from the 116 GiB iGPU runs is the plausible cause). Every `podman` +- **Incident, same evening:** the rootless podman pause process died — trigger + NOT established (no OOM kill in `journalctl -k` for that window, so "the + 116 GiB runs squeezed it out" is a guess, not a finding). Every `podman` command then failed (`invalid internal status … could not find any running process`), every container start failed on aardvark-dns binding `10.89.x.1:53` (the rootless netns holding the bridges was gone), and all 28 quadlets looped.