Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

7.8 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Dankclaw Swap Configuration

Host: Dankclaw (192.168.0.143) — Filesystem: ext4 on /dev/nvme0n1p2

Item	Value
Swapfile	`/swap.img` (68,719,476,736 B = 64 GiB, owner root 0600, mtime 2026-09-12 21:56 — `ls -la /swap.img`)
Size	64 GiB (raised from 8 GiB on 2026-09-12); `/proc/swaps` reports 67,108,860 KiB
Type / priority	`file`, priority -1 — the only swap device on the host (`swapon --noheadings` 2026-10-18: `/swap.img file 64G 187.6M -1`)
Filesystem	ext4 on `/dev/nvme0n1p2` (`df -T /` 2026-10-18) — swapfiles fully supported, no btrfs `nocow` handling needed
Allocation	`fallocate` (preallocated, hole-free; 530 extents — `sudo filefrag -v /swap.img` → "530 extents found", 2026-10-18)
Persistence	`/etc/fstab` line 12: `/swap.img none swap sw 0 0` (`grep -n -i swap /etc/fstab` 2026-10-18)
systemd unit	`swap.img.swap` — `LoadState=loaded`, `FragmentPath=/run/systemd/generator/swap.img.swap`, `UnitFileState=generated`, `is-active` → `active` (2026-10-18)
`vm.swappiness`	`60` (distro default, unchanged — `sysctl vm.swappiness` 2026-10-18)
zswap	disabled (`/sys/module/zswap/parameters/enabled` = `N`, 2026-10-18)
Swap in use	187 MiB of 63 GiB (`free -h` 2026-10-18)
Free on `/`	424 GiB of 1.9 TiB, 77 % used (`df -T /` 2026-10-18). The 748 GiB that stood right after the 2026-09-12 resize is gone — ordinary fleet writes since then

The unit is generated from fstab, not a hand-written unit file. fstab is the only control point; systemctl daemon-reload re-reads it. Nothing in this repository references the swapfile — it is host-level state, documented here so it is not invisible.

Why 64 GiB

Dankclaw is shared. The 2026-09-12 decision measured frank-dragoon alone holding 103.5 GiB of the 121.2 GiB host, leaving roughly 7.1 GiB for the then-28 co-tenant containers whose working set was ~4.3 GiB. How that total is split between weights, KV pool and working memory is the engine's own ledger — it lives on frank-dragoon. That is a decision-time reading, not today's: on 2026-10-18 the host reports 91 GiB used / 29 GiB available of 121 GiB (free -h), 30 containers are up (podman ps -q | wc -l), and the frank-dragoon-halogen container accounts for 70.33 GiB (podman stats --no-stream, same day). The engine itself warned at the time that when the headroom runs out, frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with no disk activity and no output". Swap is the valve that turns an OOM kill into slow paging.

Swap does not relieve the pinned-weights problem — see the caveat below. It protects the co-tenants and buys headroom for burst allocation.

The safe resize ordering

You cannot grow an active swapfile; it must be swapped off first. On a host like this, never swapoff before the replacement is online. The old 8 GiB held 4 GiB of live pages, and MemAvailable read 73 GiB while only ~5 GiB was genuinely free — the pinned model weights are counted as reclaimable page cache, so the headline number is a lie by roughly the size of the model (see frank-dragoon). Swapping off into that apparent space, with nothing else to receive the pages, is how you invite the OOM killer.

The ordering that avoids it — create and enable the new swap first, so the pages have somewhere to go:

# 1. Preallocate + format + enable the replacement, old swap still live
sudo fallocate -l 64G /swap2.img
sudo chmod 600 /swap2.img
sudo mkswap /swap2.img          # also rejects files with holes
sudo swapon /swap2.img          # total swap now 72 GiB

# 2. Only now drain the old one — pages migrate into /swap2.img, not into RAM
sudo swapoff /swap.img          # took 65 s for 4 GiB

# 3. Consolidate onto the conventional path so fstab needs no edit
sudo swapoff /swap2.img         # 725 MiB left, trivial
sudo mkdir -p ~/.AGENTTRASH/20260912-swapfile-8g
sudo mv /swap.img ~/.AGENTTRASH/20260912-swapfile-8g/swap.img
sudo mv /swap2.img /swap.img
sudo swapon /swap.img

Step 3 is chained tightly because it leaves the host with zero swap for a few seconds. Do not leave that window open.

mkswap succeeding is itself a check: it refuses swapfiles containing holes, so a fallocate that produced a sparse file fails here rather than at 3 a.m. under memory pressure.

Verification

swapon --show=NAME,TYPE,SIZE,USED,PRIO   # /swap.img file 64G 187.6M -1 (2026-10-18)
systemctl is-active swap.img.swap        # active; FragmentPath=/run/systemd/generator/swap.img.swap
grep -n swap /etc/fstab                  # line 12 intact -> boot persistence
free -h | tail -2                        # Swap: 63Gi total, 187Mi used (2026-10-18)

Then confirm the inference service is unaffected — a completion, not just /health:

curl -s http://127.0.0.1:7770/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"frank-dragoon","messages":[{"role":"user","content":"Reply with exactly: SWAP64-OK"}],"max_tokens":48,"reasoning_effort":"low"}'

As run 2026-09-12: finish_reason: stop, SWAP64-OK, 28 containers up, frank-dragoon health=200 throughout.

Caveats

Swap cannot help the pinned weights. frank-dragoon's weights are pinned deliberately; paging them is a stall, not a relief. More swap does not make a larger KV pool affordable — see the reserve/pool ceiling in frank-dragoon.
vm.swappiness=60 is untested against this workload. Under real pressure the kernel will happily page anonymous memory, and frank-dragoon's KV pool and working set are anonymous. Swapping the KV pool would be far worse than the OOM kill it prevents. If the box ever starts thrashing, lowering swappiness (or capping the engine's cgroup) is the knob, not more swap.
free still lies after the change. On 2026-09-12 MemAvailable read ~70 GiB while the model was loaded; on 2026-10-18 it reads 29 GiB with 90 GiB sitting in buff/cache (free -h). The headline number moves with the payload; trust the engine's host memory left for everything else line, not free.
Filesystem prerequisite (other hosts). ext4 and xfs take swapfiles directly. btrfs needs chattr +C (NOCOW), no compression, and no snapshots in the swapfile path — check df -T before planning, or mkswap will reject the file.
Renaming an active swapfile works (inode-based) but leaves a stale path in /proc/swaps — swap it off first, then rename, then swapon (which is exactly what the step-3 consolidation does). Migration is observable while it runs: in the 2026-09-12 run swapoff of 4 GiB took 65 s and used-RAM rose only ~3 GiB (pages touched during migration), the rest relocating on disk. If used-RAM climbs toward the whole swapped amount, let it finish rather than aborting.

Trash

The previous 8 GiB /swap.img (dated 2026-07-21) was staged, not deleted, at dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img (8.1 GiB). That staging directory no longer exists — ls /home/dank/.AGENTTRASH on 2026-10-18 lists only 20261005-frank-small-ubatch-ab, 20261006-frank-large-tuning, 20261007-openwebui-modelpicker, 20261010-fleet-rename, 20261017-bonsai-scratch, 20261017-prism-build, 20261017-v2abl-trial, and there is no /root/.AGENTTRASH. The old swapfile has therefore been purged; nothing on the host still holds it.

7.8 KiB Raw Blame History