live: heartbeat sync 2026-10-30T16:20:49-07:00 (Refs #11)

agent/managed-skills/dankserver-disk-census/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: dankserver-disk-census
+description: "Analyze what fills dankserver's root filesystem (excluding /mnt/entropy) and price reclaim safely: du/findmnt method, podman+docker accounting, Plex DB backups, orphan KV spill, ext4 reserve and the calibration space gate."
+---
+
+# Dankserver root-filesystem census
+
+Use when asked "what is using disk on dankserver", why `/` is full, or what must be freed before a
+calibration/convert window. Read-only first; recommend, never delete without Frank's explicit OK.
+
+## Shape of the box (verify, do not assume)
+
+`/` is `/dev/nvme0n1p2` ext4 (~1.8T). `/mnt/entropy` is a separate ~91T LUKS/ext4 volume; when the
+task says exclude it, `du -x` already stays on the root filesystem. `/var/lib/docker` and
+`/var/lib/plexmediaserver` are on `/`, not entropy.
+
+```bash
+ssh dankserver 'df -hT -x tmpfs -x devtmpfs -x squashfs; findmnt -t ext4,xfs,btrfs,zfs'
+ssh dankserver 'sudo du -x -d1 -h / 2>/dev/null | sort -h | tail -20'
+ssh dankserver 'sudo du -x -d2 -h /home 2>/dev/null | sort -h | tail -30'
+ssh dankserver 'sudo du -x -d2 -h /var  2>/dev/null | sort -h | tail -20'
+```
+
+`du` is device usage (st_blocks), so sparse files read small; `df` is the truth for free space.
+A full top-level scan of `/` takes ~2 min — background it and keep working.
+
+## Where the bytes actually live
+
+- `/home/dank/models`: the vendor HF checkpoint **and** its `.rad` conversion are two representations
+  of the same weights (each ~336G for Qwen3.8-Flash-Next) plus the live serving `.rad`. All can be
+  legitimately present; check `scripts/frank-archon.env` for the live payload before calling any of
+  them dead.
+- `/home/dank/.local/share/containers`: rootless podman storage is usually the largest non-model
+  consumer. Do not estimate it with `du` alone.
+- `/home/dank/radiance-state/kvcache`: disk KV spill (`blocks.bin`, `ckpts.bin`). It is orphaned
+  whenever the running container does not mount it — check
+  `podman inspect frank-archon --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'`
+  (rootless, `XDG_RUNTIME_DIR=/run/user/$(id -u)`). The unit is a root unit but the container is
+  rootless; `sudo podman ps` looks empty even while it serves.
+- `/home/dank/corpora` vs `/home/dank/campaign/<name>`: there may be an OLD campaign's grams/KLD
+  reference beside the current one. Identify a KLD reference's base by its header, not its date:
+  `head -c 600 corpora/kld-ref/kld.json` → `"model": "/models/rad/....rad"`.
+- `/var/lib/plexmediaserver/.../Plug-in Support/Databases`: Plex keeps dated DB backups; four stale
+  sets can be tens of GiB. `Metadata/` and `Media/` are thumbnails and chapter art.
+- `/opt/vllm`, `/opt/llama.cpp`, `~/.local/bin/llama*`: pre-radiance build leftovers; confirm no unit
+  references them (`grep -RIl /opt/vllm /etc/systemd/system /home/dank/.config/systemd`).
+
+## Container accounting
+
+```bash
+XDG_RUNTIME_DIR=/run/user/$(id -u) podman system df
+XDG_RUNTIME_DIR=/run/user/$(id -u) podman images --sort size --format '{{.Size}}\t{{.Repository}}:{{.Tag}}\t{{.ID}}'
+XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps --format '{{.Image}}' | sort -u   # in use
+sudo docker system df; sudo docker ps -aq | wc -l; systemctl is-enabled docker
+```
+
+- `podman system df`'s **reclaimable** is the authoritative figure. Summing per-image `.Size` over
+  dangling images double-counts shared layers and can read several times larger than the disk.
+- Docker may be `disabled` with zero containers yet hold ~120G of images from the vLLM era: full
+  reclaim candidate.
+- Keep the radiance pins and the media-stack images; the big unused ones are vllm/rocm/fp030/harbor.
+
+## Free-space levers and their traps
+
+- **Trash is space-neutral on the same filesystem.** Moving a 336G payload to `~/.AGENTTRASH` frees
+  nothing until the trash is purged; say so before recommending it.
+- **ext4 reserves 5 %** (`tune2fs -l /dev/nvme0n1p2 | grep -i 'Reserved block count'`, × block size).
+  `df` Avail excludes it, and the campaign runs as `dank`, so the reserve silently blocks non-root
+  writers while root still has room. Lowering to 1 % releases the reserve with no deletion, but it is
+  the fragmentation cushion on a full disk — last resort, never the first move.
+- **The calibration window has a 60 GiB gate** (`campaign/*/window-when-space.sh`,
+  `Q38_CALIB_MIN_FREE_MIB=61440`). A failed tap keeps the BF16 container on disk; banked-but-not-purged
+  is not free.
+- `uv`'s archive cache (`~/.cache/uv/archive-v0`) is routinely tens of GiB and safe to `uv cache prune`.
+
+## Reporting
+
+Rank by reclaim size, then risk: orphan KV spill and Docker images first; stale Plex DB backups and
+old-campaign corpora next; `/opt`/binaries/journal/snaps last. Give exact path, size, evidence that
+nothing uses it, and what breaks if it is wrong. Never execute a prune or delete in the census turn —
+Frank approves each removal, and the repo deletion law (trash, never `rm`) still applies to anything
+that is a record rather than a cache.