--- name: dankserver-disk-census description: "Analyze what fills dankserver's root filesystem (excluding /mnt/entropy) and price reclaim safely: du/findmnt method, podman+docker accounting, Plex DB backups, orphan KV spill, ext4 reserve and the calibration space gate." --- # Dankserver root-filesystem census Use when asked "what is using disk on dankserver", why `/` is full, or what must be freed before a calibration/convert window. Read-only first; recommend, never delete without Frank's explicit OK. ## Shape of the box (verify, do not assume) `/` is `/dev/nvme0n1p2` ext4 (~1.8T). `/mnt/entropy` is a separate ~91T LUKS/ext4 volume; when the task says exclude it, `du -x` already stays on the root filesystem. `/var/lib/docker` and `/var/lib/plexmediaserver` are on `/`, not entropy. ```bash ssh dankserver 'df -hT -x tmpfs -x devtmpfs -x squashfs; findmnt -t ext4,xfs,btrfs,zfs' ssh dankserver 'sudo du -x -d1 -h / 2>/dev/null | sort -h | tail -20' ssh dankserver 'sudo du -x -d2 -h /home 2>/dev/null | sort -h | tail -30' ssh dankserver 'sudo du -x -d2 -h /var 2>/dev/null | sort -h | tail -20' ``` `du` is device usage (st_blocks), so sparse files read small; `df` is the truth for free space. A full top-level scan of `/` takes ~2 min — background it and keep working. ## Where the bytes actually live - `/home/dank/models`: the vendor HF checkpoint **and** its `.rad` conversion are two representations of the same weights (each ~336G for Qwen3.8-Flash-Next) plus the live serving `.rad`. All can be legitimately present; check `scripts/frank-archon.env` for the live payload before calling any of them dead. - `/home/dank/.local/share/containers`: rootless podman storage is usually the largest non-model consumer. Do not estimate it with `du` alone. - `/home/dank/radiance-state/kvcache`: disk KV spill (`blocks.bin`, `ckpts.bin`). It is orphaned whenever the running container does not mount it — check `podman inspect frank-archon --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'` (rootless, `XDG_RUNTIME_DIR=/run/user/$(id -u)`). The unit is a root unit but the container is rootless; `sudo podman ps` looks empty even while it serves. - `/home/dank/corpora` vs `/home/dank/campaign/`: there may be an OLD campaign's grams/KLD reference beside the current one. Identify a KLD reference's base by its header, not its date: `head -c 600 corpora/kld-ref/kld.json` → `"model": "/models/rad/....rad"`. - `/var/lib/plexmediaserver/.../Plug-in Support/Databases`: Plex keeps dated DB backups; four stale sets can be tens of GiB. `Metadata/` and `Media/` are thumbnails and chapter art. - `/opt/vllm`, `/opt/llama.cpp`, `~/.local/bin/llama*`: pre-radiance build leftovers; confirm no unit references them (`grep -RIl /opt/vllm /etc/systemd/system /home/dank/.config/systemd`). ## Container accounting ```bash XDG_RUNTIME_DIR=/run/user/$(id -u) podman system df XDG_RUNTIME_DIR=/run/user/$(id -u) podman images --sort size --format '{{.Size}}\t{{.Repository}}:{{.Tag}}\t{{.ID}}' XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps --format '{{.Image}}' | sort -u # in use sudo docker system df; sudo docker ps -aq | wc -l; systemctl is-enabled docker ``` - `podman system df`'s **reclaimable** is the authoritative figure. Summing per-image `.Size` over dangling images double-counts shared layers and can read several times larger than the disk. - Docker may be `disabled` with zero containers yet hold ~120G of images from the vLLM era: full reclaim candidate. - Keep the radiance pins and the media-stack images; the big unused ones are vllm/rocm/fp030/harbor. ## Free-space levers and their traps - **Trash is space-neutral on the same filesystem.** Moving a 336G payload to `~/.AGENTTRASH` frees nothing until the trash is purged; say so before recommending it. - **ext4 reserves 5 %** (`tune2fs -l /dev/nvme0n1p2 | grep -i 'Reserved block count'`, × block size). `df` Avail excludes it, and the campaign runs as `dank`, so the reserve silently blocks non-root writers while root still has room. Lowering to 1 % releases the reserve with no deletion, but it is the fragmentation cushion on a full disk — last resort, never the first move. - **The calibration window has a 60 GiB gate** (`campaign/*/window-when-space.sh`, `Q38_CALIB_MIN_FREE_MIB=61440`). A failed tap keeps the BF16 container on disk; banked-but-not-purged is not free. - `uv`'s archive cache (`~/.cache/uv/archive-v0`) is routinely tens of GiB and safe to `uv cache prune`. ## Reporting Rank by reclaim size, then risk: orphan KV spill and Docker images first; stale Plex DB backups and old-campaign corpora next; `/opt`/binaries/journal/snaps last. Give exact path, size, evidence that nothing uses it, and what breaks if it is wrong. Never execute a prune or delete in the census turn — Frank approves each removal, and the repo deletion law (trash, never `rm`) still applies to anything that is a record rather than a cache.