docs(dankclaw): record the 64 GiB host swapfile and its safe resize ordering
Raised /swap.img from 8 GiB to 64 GiB on Dankclaw. Host-level state with no repo file, so it was invisible to the repo until now: fstab line 12 drives the generated swap.img.swap unit, ext4, fallocated hole-free, swappiness 60, zswap off. 748 GiB still free on /. The reusable part is the ordering. The old file held 4 GiB of live pages and MemAvailable read 73 GiB while only ~5 GiB was genuinely free -- the pinned model weights are counted as reclaimable page cache, so the headline lies by the size of the model. Swapping off before the replacement is online would have pushed those pages into RAM that does not exist. Creating and enabling the new swap first let them migrate instead: swapoff took 65 s and the host never wobbled. New page serverAppConfigs-docs/Dankclaw/Swap Configuration.md, linked from the index, plus a pointer in the AGENTS.md memory-model section so nobody reasons about headroom without checking swapon. Old 8 GiB file staged at dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/ pending approval to purge.
This commit is contained in:
Frank
@@ -662,6 +662,17 @@ GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or
|
||||
`dmidecode` for the GPU budget. They show the CPU-side carve-out. This is
|
||||
intentional. Do not change it.
|
||||
|
||||
Host swap on Dankclaw is a **64 GiB** file at `/swap.img` (raised from 8 GiB
|
||||
on 2026-09-12), fstab-driven via the generated `swap.img.swap` unit. It is
|
||||
host-level state with no repo file, so it is invisible to `deploy.sh` — check
|
||||
`swapon --show` before reasoning about headroom. It protects the co-tenants,
|
||||
not frank-large: the ~68 GiB of pinned weights are not helped by swap, and
|
||||
swapping the KV pool would be worse than the OOM it prevents. If you ever
|
||||
resize it, **create and `swapon` the replacement before `swapoff` of the old
|
||||
one** — otherwise the live pages must land in RAM, and `MemAvailable` overstates
|
||||
what is there by the size of the model. Full procedure and caveats:
|
||||
[[Swap Configuration]] (`serverAppConfigs-docs/Dankclaw/Swap Configuration.md`).
|
||||
|
||||
## Why
|
||||
|
||||
Direct edits on the remote drift from version control. Future agents will not
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
type: configuration
|
||||
title: Dankclaw Swap Configuration
|
||||
description: The host swapfile on Dankclaw — 64 GiB at /swap.img on ext4, fstab-driven via the generated swap.img.swap unit. Covers why it was raised from 8 GiB, the safe resize ordering that avoids an OOM on a host whose MemAvailable is inflated by pinned model weights, and the swappiness caveat.
|
||||
tags:
|
||||
- dankclaw
|
||||
- host
|
||||
- memory
|
||||
- swap
|
||||
timestamp: 2026-09-12
|
||||
---
|
||||
|
||||
# Dankclaw Swap Configuration
|
||||
|
||||
**Host:** Dankclaw (192.168.0.143) — **Filesystem:** ext4 on `/dev/nvme0n1p2`
|
||||
|
||||
| Item | Value |
|
||||
| --- | --- |
|
||||
| Swapfile | `/swap.img` |
|
||||
| Size | **64 GiB** (raised from 8 GiB on 2026-09-12) |
|
||||
| Filesystem | ext4 — swapfiles fully supported, no btrfs `nocow` handling needed |
|
||||
| Allocation | `fallocate` (preallocated, hole-free; 530 extents) |
|
||||
| Persistence | `/etc/fstab` line 12: `/swap.img none swap sw 0 0` |
|
||||
| systemd unit | `swap.img.swap` — `loaded (/etc/fstab; generated)`, `active` |
|
||||
| `vm.swappiness` | `60` (distro default, unchanged) |
|
||||
| zswap | disabled (`/sys/module/zswap/parameters/enabled` = `N`) |
|
||||
| Free on `/` after | 748 GiB of 1.9 TiB |
|
||||
|
||||
The unit is **generated from fstab**, not a hand-written unit file. fstab is the
|
||||
only control point; `systemctl daemon-reload` re-reads it. Nothing in this
|
||||
repository references the swapfile — it is host-level state, documented here so
|
||||
it is not invisible.
|
||||
|
||||
## Why 64 GiB
|
||||
|
||||
Dankclaw is shared. `frank-large` alone holds **103.5 GiB of the 121.2 GiB**
|
||||
host (`68.0 weights + 14.4 KV pool + 21.1 working`), leaving roughly
|
||||
**7.1 GiB** for 28 co-tenant containers whose working set is ~4.3 GiB. The
|
||||
engine itself warns that when that runs out, frank-large and its neighbours
|
||||
"can stall for minutes at a time at full CPU with no disk activity and no
|
||||
output". Swap is the valve that turns an OOM kill into slow paging.
|
||||
|
||||
Swap does **not** relieve the pinned-weights problem — see the caveat below.
|
||||
It protects the *co-tenants* and buys headroom for burst allocation.
|
||||
|
||||
## The safe resize ordering
|
||||
|
||||
You cannot grow an active swapfile; it must be swapped off first. On a host
|
||||
like this, **never `swapoff` before the replacement is online.** The old 8 GiB
|
||||
held 4 GiB of live pages, and `MemAvailable` read 73 GiB while only ~5 GiB was
|
||||
genuinely free — the pinned model weights are counted as reclaimable page
|
||||
cache, so the headline number is a lie by roughly the size of the model
|
||||
(see [[halogen-flash-server]]). Swapping off into that apparent space, with
|
||||
nothing else to receive the pages, is how you invite the OOM killer.
|
||||
|
||||
The ordering that avoids it — create and enable the new swap **first**, so the
|
||||
pages have somewhere to go:
|
||||
|
||||
```bash
|
||||
# 1. Preallocate + format + enable the replacement, old swap still live
|
||||
sudo fallocate -l 64G /swap2.img
|
||||
sudo chmod 600 /swap2.img
|
||||
sudo mkswap /swap2.img # also rejects files with holes
|
||||
sudo swapon /swap2.img # total swap now 72 GiB
|
||||
|
||||
# 2. Only now drain the old one — pages migrate into /swap2.img, not into RAM
|
||||
sudo swapoff /swap.img # took 65 s for 4 GiB
|
||||
|
||||
# 3. Consolidate onto the conventional path so fstab needs no edit
|
||||
sudo swapoff /swap2.img # 725 MiB left, trivial
|
||||
sudo mkdir -p ~/.AGENTTRASH/20260912-swapfile-8g
|
||||
sudo mv /swap.img ~/.AGENTTRASH/20260912-swapfile-8g/swap.img
|
||||
sudo mv /swap2.img /swap.img
|
||||
sudo swapon /swap.img
|
||||
```
|
||||
|
||||
Step 3 is chained tightly because it leaves the host with **zero swap** for a
|
||||
few seconds. Do not leave that window open.
|
||||
|
||||
`mkswap` succeeding is itself a check: it refuses swapfiles containing holes,
|
||||
so a `fallocate` that produced a sparse file fails here rather than at 3 a.m.
|
||||
under memory pressure.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
swapon --show=NAME,TYPE,SIZE,USED,PRIO # /swap.img file 64G
|
||||
systemctl is-active swap.img.swap # active
|
||||
grep swap /etc/fstab # line 12 intact -> boot persistence
|
||||
free -h | tail -2 # Swap: 63Gi total
|
||||
```
|
||||
|
||||
Then confirm the inference service is unaffected — a completion, not just
|
||||
`/health`:
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:7770/v1/chat/completions \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"model":"frank-large","messages":[{"role":"user","content":"Reply with exactly: SWAP64-OK"}],"max_tokens":48,"reasoning_effort":"low"}'
|
||||
```
|
||||
|
||||
As run 2026-09-12: `finish_reason: stop`, `SWAP64-OK`, 28 containers up,
|
||||
frank-large `health=200` throughout.
|
||||
|
||||
## Caveats
|
||||
|
||||
- **Swap cannot help the pinned weights.** frank-large's ~68 GiB of weights are
|
||||
held deliberately; paging them is a stall, not a relief. More swap does not
|
||||
make the doubled KV pool affordable — see the reserve/pool ceiling in
|
||||
[[halogen-flash-server]].
|
||||
- **`vm.swappiness=60` is untested against this workload.** Under real
|
||||
pressure the kernel will happily page anonymous memory, and frank-large's KV
|
||||
pool and working set *are* anonymous. Swapping the KV pool would be far worse
|
||||
than the OOM kill it prevents. If the box ever starts thrashing, lowering
|
||||
swappiness (or capping the engine's cgroup) is the knob, not more swap.
|
||||
- **`free` still lies after the change.** `MemAvailable` reads ~70 GiB while
|
||||
the model is loaded. Trust the engine's `host memory left for everything
|
||||
else` line, not `free`.
|
||||
|
||||
## Trash
|
||||
|
||||
The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at
|
||||
`dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). Purge
|
||||
only on explicit approval.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[halogen-flash-server]] — the memory model that motivated this
|
||||
- [[Dankclaw System Specs]] — the Strix Halo box
|
||||
- [[Dankclaw Improvements]] — other host-level changes
|
||||
@@ -144,6 +144,10 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box hardware
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large)
|
||||
- [[Dankclaw Improvements]] — host-level changes on the Strix Halo box
|
||||
- [[Swap Configuration]] — the 64 GiB `/swap.img` host swapfile (raised from
|
||||
8 GiB 2026-09-12): fstab-driven `swap.img.swap` unit, the safe resize
|
||||
ordering that drains the old file into fresh swap instead of into a host
|
||||
whose `MemAvailable` lies by the size of the model, and the swappiness caveat
|
||||
- [[frank-npu]] — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11):
|
||||
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the
|
||||
retirement record, including the IOMMU/SVA err=-19 root cause and
|
||||
|
||||
Reference in New Issue
Block a user