83 lines
4.9 KiB
Markdown
83 lines
4.9 KiB
Markdown
---
|
||
name: dankserver-disk-census
|
||
description: "Analyze what fills dankserver's root filesystem (excluding /mnt/entropy) and price reclaim safely: du/findmnt method, podman+docker accounting, Plex DB backups, orphan KV spill, ext4 reserve and the calibration space gate."
|
||
---
|
||
|
||
# Dankserver root-filesystem census
|
||
|
||
Use when asked "what is using disk on dankserver", why `/` is full, or what must be freed before a
|
||
calibration/convert window. Read-only first; recommend, never delete without Frank's explicit OK.
|
||
|
||
## Shape of the box (verify, do not assume)
|
||
|
||
`/` is `/dev/nvme0n1p2` ext4 (~1.8T). `/mnt/entropy` is a separate ~91T LUKS/ext4 volume; when the
|
||
task says exclude it, `du -x` already stays on the root filesystem. `/var/lib/docker` and
|
||
`/var/lib/plexmediaserver` are on `/`, not entropy.
|
||
|
||
```bash
|
||
ssh dankserver 'df -hT -x tmpfs -x devtmpfs -x squashfs; findmnt -t ext4,xfs,btrfs,zfs'
|
||
ssh dankserver 'sudo du -x -d1 -h / 2>/dev/null | sort -h | tail -20'
|
||
ssh dankserver 'sudo du -x -d2 -h /home 2>/dev/null | sort -h | tail -30'
|
||
ssh dankserver 'sudo du -x -d2 -h /var 2>/dev/null | sort -h | tail -20'
|
||
```
|
||
|
||
`du` is device usage (st_blocks), so sparse files read small; `df` is the truth for free space.
|
||
A full top-level scan of `/` takes ~2 min — background it and keep working.
|
||
|
||
## Where the bytes actually live
|
||
|
||
- `/home/dank/models`: the vendor HF checkpoint **and** its `.rad` conversion are two representations
|
||
of the same weights (each ~336G for Qwen3.8-Flash-Next) plus the live serving `.rad`. All can be
|
||
legitimately present; check `scripts/frank-archon.env` for the live payload before calling any of
|
||
them dead.
|
||
- `/home/dank/.local/share/containers`: rootless podman storage is usually the largest non-model
|
||
consumer. Do not estimate it with `du` alone.
|
||
- `/home/dank/radiance-state/kvcache`: disk KV spill (`blocks.bin`, `ckpts.bin`). It is orphaned
|
||
whenever the running container does not mount it — check
|
||
`podman inspect frank-archon --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'`
|
||
(rootless, `XDG_RUNTIME_DIR=/run/user/$(id -u)`). The unit is a root unit but the container is
|
||
rootless; `sudo podman ps` looks empty even while it serves.
|
||
- `/home/dank/corpora` vs `/home/dank/campaign/<name>`: there may be an OLD campaign's grams/KLD
|
||
reference beside the current one. Identify a KLD reference's base by its header, not its date:
|
||
`head -c 600 corpora/kld-ref/kld.json` → `"model": "/models/rad/....rad"`.
|
||
- `/var/lib/plexmediaserver/.../Plug-in Support/Databases`: Plex keeps dated DB backups; four stale
|
||
sets can be tens of GiB. `Metadata/` and `Media/` are thumbnails and chapter art.
|
||
- `/opt/vllm`, `/opt/llama.cpp`, `~/.local/bin/llama*`: pre-radiance build leftovers; confirm no unit
|
||
references them (`grep -RIl /opt/vllm /etc/systemd/system /home/dank/.config/systemd`).
|
||
|
||
## Container accounting
|
||
|
||
```bash
|
||
XDG_RUNTIME_DIR=/run/user/$(id -u) podman system df
|
||
XDG_RUNTIME_DIR=/run/user/$(id -u) podman images --sort size --format '{{.Size}}\t{{.Repository}}:{{.Tag}}\t{{.ID}}'
|
||
XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps --format '{{.Image}}' | sort -u # in use
|
||
sudo docker system df; sudo docker ps -aq | wc -l; systemctl is-enabled docker
|
||
```
|
||
|
||
- `podman system df`'s **reclaimable** is the authoritative figure. Summing per-image `.Size` over
|
||
dangling images double-counts shared layers and can read several times larger than the disk.
|
||
- Docker may be `disabled` with zero containers yet hold ~120G of images from the vLLM era: full
|
||
reclaim candidate.
|
||
- Keep the radiance pins and the media-stack images; the big unused ones are vllm/rocm/fp030/harbor.
|
||
|
||
## Free-space levers and their traps
|
||
|
||
- **Trash is space-neutral on the same filesystem.** Moving a 336G payload to `~/.AGENTTRASH` frees
|
||
nothing until the trash is purged; say so before recommending it.
|
||
- **ext4 reserves 5 %** (`tune2fs -l /dev/nvme0n1p2 | grep -i 'Reserved block count'`, × block size).
|
||
`df` Avail excludes it, and the campaign runs as `dank`, so the reserve silently blocks non-root
|
||
writers while root still has room. Lowering to 1 % releases the reserve with no deletion, but it is
|
||
the fragmentation cushion on a full disk — last resort, never the first move.
|
||
- **The calibration window has a 60 GiB gate** (`campaign/*/window-when-space.sh`,
|
||
`Q38_CALIB_MIN_FREE_MIB=61440`). A failed tap keeps the BF16 container on disk; banked-but-not-purged
|
||
is not free.
|
||
- `uv`'s archive cache (`~/.cache/uv/archive-v0`) is routinely tens of GiB and safe to `uv cache prune`.
|
||
|
||
## Reporting
|
||
|
||
Rank by reclaim size, then risk: orphan KV spill and Docker images first; stale Plex DB backups and
|
||
old-campaign corpora next; `/opt`/binaries/journal/snaps last. Give exact path, size, evidence that
|
||
nothing uses it, and what breaks if it is wrong. Never execute a prune or delete in the census turn —
|
||
Frank approves each removal, and the repo deletion law (trash, never `rm`) still applies to anything
|
||
that is a record rather than a cache.
|