command-center/agent/managed-skills/dankserver-space-reclaim/SKILL.md

name, description

name	description
dankserver-space-reclaim	Execute approved low-risk disk reclaim on dankserver's root fs: prove targets unreferenced, orphan KV first, dangling images, Docker prune, Plex DB backups; the campaign space watcher fires the moment the gate opens.

Dankserver root-fs reclaim (approved low-risk sets)

Run only with the user's explicit approval for the named set. Deletion is the point: moving a cache to .AGENTTRASH on the same filesystem frees nothing, so say that before proposing it. Never touch /mnt/entropy (separate 90T volume) and never delete /home/dank/models/rad/qwen38-bf16.rad or campaign/** while the qwen38 campaign is unfinished.

Census (live, never from docs)

ssh dankserver 'df -hT -x tmpfs -x devtmpfs; findmnt -o TARGET,SOURCE,FSTYPE,SIZE,USED,AVAIL -t ext4,xfs,btrfs'
ssh dankserver 'sudo du -x -d1 -h / 2>/dev/null | sort -h | tail -15'

-x keeps /mnt/entropy out automatically (separate mount). Then drill /home and /var with sudo du -x -d2. Typical 2026-10 shape: /home ≈ 1.4T (models 785G, .local/share/containers 334G, radiance-state 69G, corpora 64G, campaign 45G), /var ≈ 215G (docker 119G, plex 80G), /opt 17G. Verify live; sizes are disposable.

Do NOT grep -R across /home/dank/Containers or ~/Projects — jellyfin cache JSON makes it run for minutes. Scope find -maxdepth 3 -iname '*harbor*' and check unit dirs only.

Prove each target unreferenced before deleting

• sudo lsof <paths> (fuse warnings are noise) and, for podman image IDs, cross-check podman ps -a --format '{{.ImageID}}'; podman image prune keeps an in-use untagged image, but confirm the container survives.
• Rootless vs rootful matters: frank-archon runs as rootless podman under dank even though its unit is a root unit. Check both sudo podman ps -a and XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps -a, including stopped containers, and inspect their mounts (podman inspect -f '{{range .Mounts}}{{.Source}} {{end}}').
• grep -RIl '<path>' /etc/systemd/system /home/dank/.config/systemd must be empty.
• A running archon mounts only /models, radiance-state/tune and the template — not radiance-state/kvcache, so stale blocks.bin/ckpts.bin are genuinely orphaned.

Reclaim ladder (safest first)

1. Orphan radiance KV spill — radiance-state/kvcache/frank-archon/{blocks.bin,ckpts.bin} (measured 69G, last write weeks old, no handle). sudo rm -f both. This alone can clear the campaign gate.
2. Docker — only if systemctl is-enabled docker is disabled and docker ps -aq is empty: sudo docker system prune -a -f (measured ~104GB; volumes are NOT removed, keep it that way). Verify docker volume ls still lists the volumes.
3. Podman dangling only — podman image prune -f (default dangling, no -a without approval). Measured 33 removed / ~62G freed; one dangling ID is the running calibration image and is skipped. The remote shell is zsh: unquoted $ids does not word-split, so podman rmi $ids receives a newline-joined string and fails with invalid reference format. Pipe IDs through xargs instead: podman images --format '{{.ID}} {{.Repository}}' | grep goharbor | cut -d' ' -f1 | sort -u | xargs -r podman rmi.
4. Harbor registry images — all goharbor/* v2.13 images are ~2.1G total and safe once no harbor container/unit/compose exists (~/.harbor is a cmdh manifest, not the goharbor stack). The big harbor-hf/harbor-vllm build images live under Docker and disappear with step 2.
5. Plex DB backups — in Plug-in Support/Databases, keep the newest *.db-YYYY-MM-DD set and delete older sets (measured 3 × 4.9G blobs + 3 × 1.4G library ≈ 19G). Plex stays up; never touch the live library.db/blobs.db.
6. Later, with separate approval — Swift-era corpora/calib + corpora/kld-ref (KLD header names the base: model field), uv cache prune (~25G), /opt/vllm, /opt/llama.cpp, ~/.local/bin stale llama binaries, journal vacuum, old snaps. Tagged-but-unused podman builds (vllm/rocm/fp030) remain ~225G; removing them loses build caches, so scope them explicitly.

The campaign watcher is the side effect

campaign/qwen38-w5nl64/window-when-space.sh polls df --output=avail -B 1M <campaign>/calib every 60 s for Q38_CALIB_MIN_FREE_MIB (default 61440). The moment reclaim opens the gate it starts the documented calibration window, which stops frank-archon and boots qwen38-bf16 on port 7770. Expect production downtime and a health=000 during load; the engine is healthy once /health returns 200. Do not restart frank-archon by hand during the window — it would fight the rig for both cards.

If the window fails, read logs/window-start-*.log and /home/dank/logs/qwen38_bf16_calib.log: I stopping: draining in-flight requests + 429 the server is shutting down means the orchestrator SIGTERMed a healthy engine (tap marker state), not a crash; a retry may then fail with E startup stopped: device error while the prior process's VM memory stats ... non-zero when fini clears. Restore is automatic: the window script restores frank-archon and reports the banked/kept .rad.

Verify

df -h /; df --output=avail -B 1M /
XDG_RUNTIME_DIR=/run/user/$(id -u) podman system df
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps --format '{{.Names}} {{.Status}}'

All running containers must be untouched, archon back to 200, and the campaign gate recomputed with its own command. ext4 reserves 5% (~90G on the 1.8T root): df Avail is what the unprivileged campaign sees, so root's ability to write does not mean the gate passes; lowering the reserve (tune2fs -m 1) releases ~72G but removes the fragmentation cushion on a nearly full disk — last resort only.