95 lines
5.8 KiB
Markdown
95 lines
5.8 KiB
Markdown
---
|
||
name: dankserver-space-reclaim
|
||
description: "Execute approved low-risk disk reclaim on dankserver's root fs: prove targets unreferenced, orphan KV first, dangling images, Docker prune, Plex DB backups; the campaign space watcher fires the moment the gate opens."
|
||
---
|
||
|
||
# Dankserver root-fs reclaim (approved low-risk sets)
|
||
|
||
Run only with the user's explicit approval for the named set. Deletion is the point: moving a cache
|
||
to `.AGENTTRASH` on the same filesystem frees nothing, so say that before proposing it. Never touch
|
||
`/mnt/entropy` (separate 90T volume) and never delete `/home/dank/models/rad/qwen38-bf16.rad` or
|
||
`campaign/**` while the qwen38 campaign is unfinished.
|
||
|
||
## Census (live, never from docs)
|
||
|
||
```sh
|
||
ssh dankserver 'df -hT -x tmpfs -x devtmpfs; findmnt -o TARGET,SOURCE,FSTYPE,SIZE,USED,AVAIL -t ext4,xfs,btrfs'
|
||
ssh dankserver 'sudo du -x -d1 -h / 2>/dev/null | sort -h | tail -15'
|
||
```
|
||
|
||
`-x` keeps `/mnt/entropy` out automatically (separate mount). Then drill `/home` and `/var` with
|
||
`sudo du -x -d2`. Typical 2026-10 shape: `/home` ≈ 1.4T (models 785G, `.local/share/containers` 334G,
|
||
`radiance-state` 69G, corpora 64G, campaign 45G), `/var` ≈ 215G (docker 119G, plex 80G), `/opt` 17G.
|
||
Verify live; sizes are disposable.
|
||
|
||
Do NOT `grep -R` across `/home/dank/Containers` or `~/Projects` — jellyfin cache JSON makes it run
|
||
for minutes. Scope `find -maxdepth 3 -iname '*harbor*'` and check unit dirs only.
|
||
|
||
## Prove each target unreferenced before deleting
|
||
|
||
- `sudo lsof <paths>` (fuse warnings are noise) and, for podman image IDs, cross-check
|
||
`podman ps -a --format '{{.ImageID}}'`; `podman image prune` keeps an in-use untagged image, but
|
||
confirm the container survives.
|
||
- Rootless vs rootful matters: `frank-archon` runs as rootless podman under `dank` even though its
|
||
unit is a root unit. Check both `sudo podman ps -a` and
|
||
`XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps -a`, including stopped containers, and inspect their
|
||
mounts (`podman inspect -f '{{range .Mounts}}{{.Source}} {{end}}'`).
|
||
- `grep -RIl '<path>' /etc/systemd/system /home/dank/.config/systemd` must be empty.
|
||
- A running archon mounts only `/models`, `radiance-state/tune` and the template — **not**
|
||
`radiance-state/kvcache`, so stale `blocks.bin`/`ckpts.bin` are genuinely orphaned.
|
||
|
||
## Reclaim ladder (safest first)
|
||
|
||
1. **Orphan radiance KV spill** — `radiance-state/kvcache/frank-archon/{blocks.bin,ckpts.bin}`
|
||
(measured 69G, last write weeks old, no handle). `sudo rm -f` both. This alone can clear the
|
||
campaign gate.
|
||
2. **Docker** — only if `systemctl is-enabled docker` is `disabled` and `docker ps -aq` is empty:
|
||
`sudo docker system prune -a -f` (measured ~104GB; volumes are NOT removed, keep it that way).
|
||
Verify `docker volume ls` still lists the volumes.
|
||
3. **Podman dangling only** — `podman image prune -f` (default dangling, no `-a` without approval).
|
||
Measured 33 removed / ~62G freed; one dangling ID is the running calibration image and is skipped.
|
||
The remote shell is **zsh**: unquoted `$ids` does not word-split, so `podman rmi $ids` receives a
|
||
newline-joined string and fails with `invalid reference format`. Pipe IDs through xargs instead:
|
||
`podman images --format '{{.ID}} {{.Repository}}' | grep goharbor | cut -d' ' -f1 | sort -u | xargs -r podman rmi`.
|
||
4. **Harbor registry images** — all `goharbor/*` v2.13 images are ~2.1G total and safe once no
|
||
harbor container/unit/compose exists (`~/.harbor` is a cmdh manifest, not the goharbor stack). The
|
||
big `harbor-hf`/`harbor-vllm` build images live under Docker and disappear with step 2.
|
||
5. **Plex DB backups** — in `Plug-in Support/Databases`, keep the newest `*.db-YYYY-MM-DD` set and
|
||
delete older sets (measured 3 × 4.9G blobs + 3 × 1.4G library ≈ 19G). Plex stays up; never touch
|
||
the live `library.db`/`blobs.db`.
|
||
6. **Later, with separate approval** — Swift-era `corpora/calib` + `corpora/kld-ref` (KLD header
|
||
names the base: `model` field), `uv cache prune` (~25G), `/opt/vllm`, `/opt/llama.cpp`,
|
||
`~/.local/bin` stale llama binaries, journal vacuum, old snaps. Tagged-but-unused podman builds
|
||
(vllm/rocm/fp030) remain ~225G; removing them loses build caches, so scope them explicitly.
|
||
|
||
## The campaign watcher is the side effect
|
||
|
||
`campaign/qwen38-w5nl64/window-when-space.sh` polls
|
||
`df --output=avail -B 1M <campaign>/calib` every 60 s for `Q38_CALIB_MIN_FREE_MIB` (default 61440).
|
||
The moment reclaim opens the gate it starts the documented calibration window, which **stops
|
||
frank-archon** and boots `qwen38-bf16` on port 7770. Expect production downtime and a `health=000`
|
||
during load; the engine is healthy once `/health` returns 200. Do not restart frank-archon by hand
|
||
during the window — it would fight the rig for both cards.
|
||
|
||
If the window fails, read `logs/window-start-*.log` and `/home/dank/logs/qwen38_bf16_calib.log`:
|
||
`I stopping: draining in-flight requests` + 429 `the server is shutting down` means the orchestrator
|
||
SIGTERMed a healthy engine (tap marker state), not a crash; a retry may then fail with
|
||
`E startup stopped: device error` while the prior process's `VM memory stats ... non-zero when fini`
|
||
clears. Restore is automatic: the window script restores frank-archon and reports the banked/kept
|
||
`.rad`.
|
||
|
||
## Verify
|
||
|
||
```sh
|
||
df -h /; df --output=avail -B 1M /
|
||
XDG_RUNTIME_DIR=/run/user/$(id -u) podman system df
|
||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
|
||
XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps --format '{{.Names}} {{.Status}}'
|
||
```
|
||
|
||
All running containers must be untouched, archon back to 200, and the campaign gate recomputed with
|
||
its own command. ext4 reserves 5% (~90G on the 1.8T root): `df` Avail is what the unprivileged
|
||
campaign sees, so root's ability to write does not mean the gate passes; lowering the reserve
|
||
(`tune2fs -m 1`) releases ~72G but removes the fragmentation cushion on a nearly full disk — last
|
||
resort only.
|