Frank
4b82f35354
Tracked the harness artifacts accumulated on blankpad (blank@), matching the existing convention for the --home-dank-* memory stores and the 36 already-tracked managed-skills. managed-skills (9): authentik-proxy-provider-app-link, dankclaw-guard-chain-layer-diagnosis, e2e-arr-rawip-reachability-triage, halogen-kv-pool-fit, lan-client-split-dns-fix, noctalia-reference, rootless-quadlet-log-retrieval, ssh-key-bootstrap-no-sshpass, swapfile-resize-safe-ordering. memories (2 stores): --home-blank-.omp-- (dotfiles_git, handy_lifecycle, input_permissions) and --home-blank-Projects-serverAppConfig-- (frank_large_quadlet). Secret sweep before staging: no PEM blocks, no credential literals. Content is fleet topology/IPs only, same class as the tracked dank stores.
63 lines
3.5 KiB
Markdown
63 lines
3.5 KiB
Markdown
---
|
|
name: swapfile-resize-safe-ordering
|
|
description: "Resize a swapfile on a host whose MemAvailable is inflated by pinned model weights or huge page cache: put the replacement swap online BEFORE swapoff so live pages migrate to disk instead of into RAM that does not exist."
|
|
---
|
|
|
|
# Resizing a swapfile without triggering the OOM killer
|
|
|
|
You cannot grow an active swapfile — it must be swapped off first. On a host that is already tight, **`swapoff` must move every live swapped page somewhere**, and if the replacement is not online yet, that somewhere is RAM.
|
|
|
|
The headline number lies. `MemAvailable` counts reclaimable page cache, and pinned/locked model weights plus ordinary cache make it read tens of GiB while only a few GiB is genuinely free. Never size a `swapoff` against `free` or `MemAvailable`.
|
|
|
|
## The ordering
|
|
|
|
Create and enable the replacement **while the old swap is still live**, so pages migrate to disk:
|
|
|
|
```bash
|
|
# 1. Replacement online first — old swap untouched
|
|
sudo fallocate -l 64G /swap2.img
|
|
sudo chmod 600 /swap2.img
|
|
sudo mkswap /swap2.img # also the hole check; refuses sparse files
|
|
sudo swapon /swap2.img # total swap = old + new
|
|
|
|
# 2. Now drain the old one — pages go into the new swap, not RAM
|
|
sudo swapoff /swap.img # minutes; watch `free` for used-RAM creep
|
|
|
|
# 3. Consolidate onto the conventional path so fstab needs no edit
|
|
sudo swapoff /swap2.img # only the residual is left
|
|
sudo mkdir -p ~/.AGENTTRASH/<date>-swapfile-<oldsize>
|
|
sudo mv /swap.img ~/.AGENTTRASH/<date>-swapfile-<oldsize>/swap.img
|
|
sudo mv /swap2.img /swap.img
|
|
sudo swapon /swap.img
|
|
```
|
|
|
|
Chain step 3 as one tight command — it leaves the host with **zero swap** for a few seconds. Do not leave that window open.
|
|
|
|
## Why each step
|
|
|
|
- **`mkswap` is a free correctness gate.** It rejects swapfiles containing holes, so a sparse `fallocate` fails loudly at format time instead of at 3 a.m. under memory pressure. Verify with `filefrag` if unsure; `unwritten` extents are allocated, not holes, and are fine.
|
|
|
|
- **Consolidating to the original path means fstab never changes**, so boot persistence carries over with no edit and no risk of a stale swap entry dropping boot to emergency. Renaming an *active* swapfile works (inode-based) but leaves a stale path in `/proc/swaps` — swap it off first, then rename, then `swapon`.
|
|
|
|
- **Migration is observable.** In the reference run, `swapoff` of 4 GiB took 65 s and used-RAM rose only ~3 GiB (pages touched during migration), the rest relocated on disk. If used-RAM climbs toward the whole swapped amount, abort reasoning and let it finish or re-enable.
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
swapon --show=NAME,TYPE,SIZE,USED,PRIO
|
|
systemctl is-active <file>.swap # e.g. swap.img.swap — generated from fstab
|
|
grep swap /etc/fstab # entry intact -> boot persistence
|
|
```
|
|
|
|
Then prove the workload survived with a **real completion**, not `/health`.
|
|
|
|
## Filesystem caveat
|
|
|
|
ext4 and xfs take swapfiles directly. **btrfs needs `chattr +C` (NOCOW)**, no compression, and no snapshots in the path — check `df -T` before planning, or the swapfile will be rejected.
|
|
|
|
## After the change
|
|
|
|
More swap does not relieve pinned memory. Check `vm.swappiness`: at the default 60 the kernel will page anonymous memory under pressure, and if a large service's hot working set is anonymous (an LLM KV pool, for instance), paging it is worse than the OOM kill it prevents. The knob for that is lower swappiness or a cgroup cap on the service — not a bigger swapfile.
|
|
|
|
Stage the old file in trash rather than deleting it, and get explicit approval before purging.
|