diff --git a/AGENTS.md b/AGENTS.md index 45ab55a..02aec46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -662,6 +662,17 @@ GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or `dmidecode` for the GPU budget. They show the CPU-side carve-out. This is intentional. Do not change it. +Host swap on Dankclaw is a **64 GiB** file at `/swap.img` (raised from 8 GiB +on 2026-09-12), fstab-driven via the generated `swap.img.swap` unit. It is +host-level state with no repo file, so it is invisible to `deploy.sh` — check +`swapon --show` before reasoning about headroom. It protects the co-tenants, +not frank-large: the ~68 GiB of pinned weights are not helped by swap, and +swapping the KV pool would be worse than the OOM it prevents. If you ever +resize it, **create and `swapon` the replacement before `swapoff` of the old +one** — otherwise the live pages must land in RAM, and `MemAvailable` overstates +what is there by the size of the model. Full procedure and caveats: +[[Swap Configuration]] (`serverAppConfigs-docs/Dankclaw/Swap Configuration.md`). + ## Why Direct edits on the remote drift from version control. Future agents will not diff --git a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md new file mode 100644 index 0000000..9d181cc --- /dev/null +++ b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md @@ -0,0 +1,130 @@ +--- +type: configuration +title: Dankclaw Swap Configuration +description: The host swapfile on Dankclaw — 64 GiB at /swap.img on ext4, fstab-driven via the generated swap.img.swap unit. Covers why it was raised from 8 GiB, the safe resize ordering that avoids an OOM on a host whose MemAvailable is inflated by pinned model weights, and the swappiness caveat. +tags: + - dankclaw + - host + - memory + - swap +timestamp: 2026-09-12 +--- + +# Dankclaw Swap Configuration + +**Host:** Dankclaw (192.168.0.143) — **Filesystem:** ext4 on `/dev/nvme0n1p2` + +| Item | Value | +| --- | --- | +| Swapfile | `/swap.img` | +| Size | **64 GiB** (raised from 8 GiB on 2026-09-12) | +| Filesystem | ext4 — swapfiles fully supported, no btrfs `nocow` handling needed | +| Allocation | `fallocate` (preallocated, hole-free; 530 extents) | +| Persistence | `/etc/fstab` line 12: `/swap.img none swap sw 0 0` | +| systemd unit | `swap.img.swap` — `loaded (/etc/fstab; generated)`, `active` | +| `vm.swappiness` | `60` (distro default, unchanged) | +| zswap | disabled (`/sys/module/zswap/parameters/enabled` = `N`) | +| Free on `/` after | 748 GiB of 1.9 TiB | + +The unit is **generated from fstab**, not a hand-written unit file. fstab is the +only control point; `systemctl daemon-reload` re-reads it. Nothing in this +repository references the swapfile — it is host-level state, documented here so +it is not invisible. + +## Why 64 GiB + +Dankclaw is shared. `frank-large` alone holds **103.5 GiB of the 121.2 GiB** +host (`68.0 weights + 14.4 KV pool + 21.1 working`), leaving roughly +**7.1 GiB** for 28 co-tenant containers whose working set is ~4.3 GiB. The +engine itself warns that when that runs out, frank-large and its neighbours +"can stall for minutes at a time at full CPU with no disk activity and no +output". Swap is the valve that turns an OOM kill into slow paging. + +Swap does **not** relieve the pinned-weights problem — see the caveat below. +It protects the *co-tenants* and buys headroom for burst allocation. + +## The safe resize ordering + +You cannot grow an active swapfile; it must be swapped off first. On a host +like this, **never `swapoff` before the replacement is online.** The old 8 GiB +held 4 GiB of live pages, and `MemAvailable` read 73 GiB while only ~5 GiB was +genuinely free — the pinned model weights are counted as reclaimable page +cache, so the headline number is a lie by roughly the size of the model +(see [[halogen-flash-server]]). Swapping off into that apparent space, with +nothing else to receive the pages, is how you invite the OOM killer. + +The ordering that avoids it — create and enable the new swap **first**, so the +pages have somewhere to go: + +```bash +# 1. Preallocate + format + enable the replacement, old swap still live +sudo fallocate -l 64G /swap2.img +sudo chmod 600 /swap2.img +sudo mkswap /swap2.img # also rejects files with holes +sudo swapon /swap2.img # total swap now 72 GiB + +# 2. Only now drain the old one — pages migrate into /swap2.img, not into RAM +sudo swapoff /swap.img # took 65 s for 4 GiB + +# 3. Consolidate onto the conventional path so fstab needs no edit +sudo swapoff /swap2.img # 725 MiB left, trivial +sudo mkdir -p ~/.AGENTTRASH/20260912-swapfile-8g +sudo mv /swap.img ~/.AGENTTRASH/20260912-swapfile-8g/swap.img +sudo mv /swap2.img /swap.img +sudo swapon /swap.img +``` + +Step 3 is chained tightly because it leaves the host with **zero swap** for a +few seconds. Do not leave that window open. + +`mkswap` succeeding is itself a check: it refuses swapfiles containing holes, +so a `fallocate` that produced a sparse file fails here rather than at 3 a.m. +under memory pressure. + +## Verification + +```bash +swapon --show=NAME,TYPE,SIZE,USED,PRIO # /swap.img file 64G +systemctl is-active swap.img.swap # active +grep swap /etc/fstab # line 12 intact -> boot persistence +free -h | tail -2 # Swap: 63Gi total +``` + +Then confirm the inference service is unaffected — a completion, not just +`/health`: + +```bash +curl -s http://127.0.0.1:7770/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"frank-large","messages":[{"role":"user","content":"Reply with exactly: SWAP64-OK"}],"max_tokens":48,"reasoning_effort":"low"}' +``` + +As run 2026-09-12: `finish_reason: stop`, `SWAP64-OK`, 28 containers up, +frank-large `health=200` throughout. + +## Caveats + +- **Swap cannot help the pinned weights.** frank-large's ~68 GiB of weights are + held deliberately; paging them is a stall, not a relief. More swap does not + make the doubled KV pool affordable — see the reserve/pool ceiling in + [[halogen-flash-server]]. +- **`vm.swappiness=60` is untested against this workload.** Under real + pressure the kernel will happily page anonymous memory, and frank-large's KV + pool and working set *are* anonymous. Swapping the KV pool would be far worse + than the OOM kill it prevents. If the box ever starts thrashing, lowering + swappiness (or capping the engine's cgroup) is the knob, not more swap. +- **`free` still lies after the change.** `MemAvailable` reads ~70 GiB while + the model is loaded. Trust the engine's `host memory left for everything + else` line, not `free`. + +## Trash + +The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at +`dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). Purge +only on explicit approval. + +## See Also + +- [[halogen-flash-server]] — the memory model that motivated this +- [[Dankclaw System Specs]] — the Strix Halo box +- [[Dankclaw Improvements]] — other host-level changes diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 037119f..eb86cd5 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -144,6 +144,10 @@ The LLM inference fleet on the two hosts. Full index: - [[Dankserver System Specs]] — the RTX 3090 box hardware - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large) - [[Dankclaw Improvements]] — host-level changes on the Strix Halo box +- [[Swap Configuration]] — the 64 GiB `/swap.img` host swapfile (raised from + 8 GiB 2026-09-12): fstab-driven `swap.img.swap` unit, the safe resize + ordering that drains the old file into fresh swap instead of into a host + whose `MemAvailable` lies by the size of the model, and the swappiness caveat - [[frank-npu]] — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11): Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the retirement record, including the IOMMU/SVA err=-19 root cause and