---
name: dankserver-space-reclaim
description: "Execute approved low-risk disk reclaim on dankserver's root fs: prove targets unreferenced, orphan KV first, dangling images, Docker prune, Plex DB backups; the campaign space watcher fires the moment the gate opens."
---

# Dankserver root-fs reclaim (approved low-risk sets)

Run only with the user's explicit approval for the named set. Deletion is the point: moving a cache
to `.AGENTTRASH` on the same filesystem frees nothing, so say that before proposing it. Never touch
`/mnt/entropy` (separate 90T volume) and never delete `/home/dank/models/rad/qwen38-bf16.rad` or
`campaign/**` while the qwen38 campaign is unfinished.

## Census (live, never from docs)

```sh
ssh dankserver 'df -hT -x tmpfs -x devtmpfs; findmnt -o TARGET,SOURCE,FSTYPE,SIZE,USED,AVAIL -t ext4,xfs,btrfs'
ssh dankserver 'sudo du -x -d1 -h / 2>/dev/null | sort -h | tail -15'
```

`-x` keeps `/mnt/entropy` out automatically (separate mount). Then drill `/home` and `/var` with
`sudo du -x -d2`. Typical 2026-10 shape: `/home` ≈ 1.4T (models 785G, `.local/share/containers` 334G,
`radiance-state` 69G, corpora 64G, campaign 45G), `/var` ≈ 215G (docker 119G, plex 80G), `/opt` 17G.
Verify live; sizes are disposable.

Do NOT `grep -R` across `/home/dank/Containers` or `~/Projects` — jellyfin cache JSON makes it run
for minutes. Scope `find -maxdepth 3 -iname '*harbor*'` and check unit dirs only.

## Prove each target unreferenced before deleting

- `sudo lsof <paths>` (fuse warnings are noise) and, for podman image IDs, cross-check
  `podman ps -a --format '{{.ImageID}}'`; `podman image prune` keeps an in-use untagged image, but
  confirm the container survives.
- Rootless vs rootful matters: `frank-archon` runs as rootless podman under `dank` even though its
  unit is a root unit. Check both `sudo podman ps -a` and
  `XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps -a`, including stopped containers, and inspect their
  mounts (`podman inspect -f '{{range .Mounts}}{{.Source}} {{end}}'`).
- `grep -RIl '<path>' /etc/systemd/system /home/dank/.config/systemd` must be empty.
- A running archon mounts only `/models`, `radiance-state/tune` and the template — **not**
  `radiance-state/kvcache`, so stale `blocks.bin`/`ckpts.bin` are genuinely orphaned.

## Reclaim ladder (safest first)

1. **Orphan radiance KV spill** — `radiance-state/kvcache/frank-archon/{blocks.bin,ckpts.bin}`
   (measured 69G, last write weeks old, no handle). `sudo rm -f` both. This alone can clear the
   campaign gate.
2. **Docker** — only if `systemctl is-enabled docker` is `disabled` and `docker ps -aq` is empty:
   `sudo docker system prune -a -f` (measured ~104GB; volumes are NOT removed, keep it that way).
   Verify `docker volume ls` still lists the volumes.
3. **Podman dangling only** — `podman image prune -f` (default dangling, no `-a` without approval).
   Measured 33 removed / ~62G freed; one dangling ID is the running calibration image and is skipped.
   The remote shell is **zsh**: unquoted `$ids` does not word-split, so `podman rmi $ids` receives a
   newline-joined string and fails with `invalid reference format`. Pipe IDs through xargs instead:
   `podman images --format '{{.ID}} {{.Repository}}' | grep goharbor | cut -d' ' -f1 | sort -u | xargs -r podman rmi`.
4. **Harbor registry images** — all `goharbor/*` v2.13 images are ~2.1G total and safe once no
   harbor container/unit/compose exists (`~/.harbor` is a cmdh manifest, not the goharbor stack). The
   big `harbor-hf`/`harbor-vllm` build images live under Docker and disappear with step 2.
5. **Plex DB backups** — in `Plug-in Support/Databases`, keep the newest `*.db-YYYY-MM-DD` set and
   delete older sets (measured 3 × 4.9G blobs + 3 × 1.4G library ≈ 19G). Plex stays up; never touch
   the live `library.db`/`blobs.db`.
6. **Later, with separate approval** — Swift-era `corpora/calib` + `corpora/kld-ref` (KLD header
   names the base: `model` field), `uv cache prune` (~25G), `/opt/vllm`, `/opt/llama.cpp`,
   `~/.local/bin` stale llama binaries, journal vacuum, old snaps. Tagged-but-unused podman builds
   (vllm/rocm/fp030) remain ~225G; removing them loses build caches, so scope them explicitly.

## The campaign watcher is the side effect

`campaign/qwen38-w5nl64/window-when-space.sh` polls
`df --output=avail -B 1M <campaign>/calib` every 60 s for `Q38_CALIB_MIN_FREE_MIB` (default 61440).
The moment reclaim opens the gate it starts the documented calibration window, which **stops
frank-archon** and boots `qwen38-bf16` on port 7770. Expect production downtime and a `health=000`
during load; the engine is healthy once `/health` returns 200. Do not restart frank-archon by hand
during the window — it would fight the rig for both cards.

If the window fails, read `logs/window-start-*.log` and `/home/dank/logs/qwen38_bf16_calib.log`:
`I stopping: draining in-flight requests` + 429 `the server is shutting down` means the orchestrator
SIGTERMed a healthy engine (tap marker state), not a crash; a retry may then fail with
`E startup stopped: device error` while the prior process's `VM memory stats ... non-zero when fini`
clears. Restore is automatic: the window script restores frank-archon and reports the banked/kept
`.rad`.

## Verify

```sh
df -h /; df --output=avail -B 1M /
XDG_RUNTIME_DIR=/run/user/$(id -u) podman system df
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps --format '{{.Names}} {{.Status}}'
```

All running containers must be untouched, archon back to 200, and the campaign gate recomputed with
its own command. ext4 reserves 5% (~90G on the 1.8T root): `df` Avail is what the unprivileged
campaign sees, so root's ability to write does not mean the gate passes; lowering the reserve
(`tune2fs -m 1`) releases ~72G but removes the fragmentation cushion on a nearly full disk — last
resort only.