docs(vault): fleet-wide fact re-verification pass (67 pages)
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
This commit is contained in:
Frank committed
1 parent
466fd477d8
commit
e7f0d93c86
67 files changed
+2619
-1152
No files matched your search
@@ -16,15 +16,17 @@ timestamp: 2026-09-12
|
||||
|
||||
| Item | Value |
|
||||
| --- | --- |
|
||||
| Swapfile | `/swap.img` |
|
||||
| Size | **64 GiB** (raised from 8 GiB on 2026-09-12) |
|
||||
| Filesystem | ext4 — swapfiles fully supported, no btrfs `nocow` handling needed |
|
||||
| Allocation | `fallocate` (preallocated, hole-free; 530 extents) |
|
||||
| Persistence | `/etc/fstab` line 12: `/swap.img none swap sw 0 0` |
|
||||
| systemd unit | `swap.img.swap` — `loaded (/etc/fstab; generated)`, `active` |
|
||||
| `vm.swappiness` | `60` (distro default, unchanged) |
|
||||
| zswap | disabled (`/sys/module/zswap/parameters/enabled` = `N`) |
|
||||
| Free on `/` after | 748 GiB of 1.9 TiB |
|
||||
| Swapfile | `/swap.img` (68,719,476,736 B = 64 GiB, owner root 0600, mtime 2026-09-12 21:56 — `ls -la /swap.img`) |
|
||||
| Size | **64 GiB** (raised from 8 GiB on 2026-09-12); `/proc/swaps` reports 67,108,860 KiB |
|
||||
| Type / priority | `file`, priority **-1** — the only swap device on the host (`swapon --noheadings` 2026-10-18: `/swap.img file 64G 187.6M -1`) |
|
||||
| Filesystem | ext4 on `/dev/nvme0n1p2` (`df -T /` 2026-10-18) — swapfiles fully supported, no btrfs `nocow` handling needed |
|
||||
| Allocation | `fallocate` (preallocated, hole-free; **530 extents** — `sudo filefrag -v /swap.img` → "530 extents found", 2026-10-18) |
|
||||
| Persistence | `/etc/fstab` line 12: `/swap.img none swap sw 0 0` (`grep -n -i swap /etc/fstab` 2026-10-18) |
|
||||
| systemd unit | `swap.img.swap` — `LoadState=loaded`, `FragmentPath=/run/systemd/generator/swap.img.swap`, `UnitFileState=generated`, `is-active` → `active` (2026-10-18) |
|
||||
| `vm.swappiness` | `60` (distro default, unchanged — `sysctl vm.swappiness` 2026-10-18) |
|
||||
| zswap | disabled (`/sys/module/zswap/parameters/enabled` = `N`, 2026-10-18) |
|
||||
| Swap in use | 187 MiB of 63 GiB (`free -h` 2026-10-18) |
|
||||
| Free on `/` | **424 GiB of 1.9 TiB, 77 % used** (`df -T /` 2026-10-18). The 748 GiB that stood right after the 2026-09-12 resize is gone — ordinary fleet writes since then |
|
||||
|
||||
The unit is **generated from fstab**, not a hand-written unit file. fstab is the
|
||||
only control point; `systemctl daemon-reload` re-reads it. Nothing in this
|
||||
@@ -33,12 +35,19 @@ it is not invisible.
|
||||
|
||||
## Why 64 GiB
|
||||
|
||||
Dankclaw is shared. `frank-dragoon` alone holds **103.5 GiB of the 121.2 GiB**
|
||||
host (`68.0 weights + 14.4 KV pool + 21.1 working`), leaving roughly
|
||||
**7.1 GiB** for 28 co-tenant containers whose working set is ~4.3 GiB. The
|
||||
engine itself warns that when that runs out, frank-dragoon and its neighbours
|
||||
"can stall for minutes at a time at full CPU with no disk activity and no
|
||||
output". Swap is the valve that turns an OOM kill into slow paging.
|
||||
Dankclaw is shared. The 2026-09-12 decision measured `frank-dragoon` alone holding
|
||||
**103.5 GiB of the 121.2 GiB** host (`68.0` weights + `14.4` KV pool + `21.1`
|
||||
working), leaving roughly **7.1 GiB** for the then-28 co-tenant containers whose
|
||||
working set was ~4.3 GiB. That is a decision-time reading, not today's: on
|
||||
2026-10-18 the host reports **91 GiB used / 29 GiB available of 121 GiB**
|
||||
(`free -h`), 30 containers are up (`podman ps -q | wc -l`), and the
|
||||
`frank-dragoon-halogen` container accounts for **70.33 GiB**
|
||||
(`podman stats --no-stream`, same day) on the current flash-server payload —
|
||||
a different model and KV shape than the one measured in September.
|
||||
The engine itself warned at the time that when the headroom runs out,
|
||||
frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with
|
||||
no disk activity and no output". Swap is the valve that turns an OOM kill into slow
|
||||
paging.
|
||||
|
||||
Swap does **not** relieve the pinned-weights problem — see the caveat below.
|
||||
It protects the *co-tenants* and buys headroom for burst allocation.
|
||||
@@ -84,10 +93,10 @@ under memory pressure.
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
swapon --show=NAME,TYPE,SIZE,USED,PRIO # /swap.img file 64G
|
||||
systemctl is-active swap.img.swap # active
|
||||
grep swap /etc/fstab # line 12 intact -> boot persistence
|
||||
free -h | tail -2 # Swap: 63Gi total
|
||||
swapon --show=NAME,TYPE,SIZE,USED,PRIO # /swap.img file 64G 187.6M -1 (2026-10-18)
|
||||
systemctl is-active swap.img.swap # active; FragmentPath=/run/systemd/generator/swap.img.swap
|
||||
grep -n swap /etc/fstab # line 12 intact -> boot persistence
|
||||
free -h | tail -2 # Swap: 63Gi total, 187Mi used (2026-10-18)
|
||||
```
|
||||
|
||||
Then confirm the inference service is unaffected — a completion, not just
|
||||
@@ -113,9 +122,10 @@ frank-dragoon `health=200` throughout.
|
||||
pool and working set *are* anonymous. Swapping the KV pool would be far worse
|
||||
than the OOM kill it prevents. If the box ever starts thrashing, lowering
|
||||
swappiness (or capping the engine's cgroup) is the knob, not more swap.
|
||||
- **`free` still lies after the change.** `MemAvailable` reads ~70 GiB while
|
||||
the model is loaded. Trust the engine's `host memory left for everything
|
||||
else` line, not `free`.
|
||||
- **`free` still lies after the change.** On 2026-09-12 `MemAvailable` read ~70 GiB
|
||||
while the model was loaded; on 2026-10-18 it reads **29 GiB** with 90 GiB sitting in
|
||||
`buff/cache` (`free -h`). The headline number moves with the payload; trust the
|
||||
engine's `host memory left for everything else` line, not `free`.
|
||||
|
||||
- **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles
|
||||
directly. btrfs needs `chattr +C` (NOCOW), no compression, and no
|
||||
@@ -131,9 +141,13 @@ frank-dragoon `health=200` throughout.
|
||||
|
||||
## Trash
|
||||
|
||||
The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at
|
||||
`dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). Purge
|
||||
only on explicit approval.
|
||||
The previous 8 GiB `/swap.img` (dated 2026-07-21) was staged, not deleted, at
|
||||
`dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). **That staging
|
||||
directory no longer exists** — `ls /home/dank/.AGENTTRASH` on 2026-10-18 lists only
|
||||
`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`,
|
||||
`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`,
|
||||
`20261017-prism-build`, `20261017-v2abl-trial`, and there is no `/root/.AGENTTRASH`.
|
||||
The old swapfile has therefore been purged; nothing on the host still holds it.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
Reference in new issue
Block a user