live: heartbeat sync 2026-10-30T16:20:49-07:00 (Refs #11)

agent/managed-skills/dankserver-space-reclaim/SKILL.md

@@ -0,0 +1,94 @@
+---
+name: dankserver-space-reclaim
+description: "Execute approved low-risk disk reclaim on dankserver's root fs: prove targets unreferenced, orphan KV first, dangling images, Docker prune, Plex DB backups; the campaign space watcher fires the moment the gate opens."
+---
+
+# Dankserver root-fs reclaim (approved low-risk sets)
+
+Run only with the user's explicit approval for the named set. Deletion is the point: moving a cache
+to `.AGENTTRASH` on the same filesystem frees nothing, so say that before proposing it. Never touch
+`/mnt/entropy` (separate 90T volume) and never delete `/home/dank/models/rad/qwen38-bf16.rad` or
+`campaign/**` while the qwen38 campaign is unfinished.
+
+## Census (live, never from docs)
+
+```sh
+ssh dankserver 'df -hT -x tmpfs -x devtmpfs; findmnt -o TARGET,SOURCE,FSTYPE,SIZE,USED,AVAIL -t ext4,xfs,btrfs'
+ssh dankserver 'sudo du -x -d1 -h / 2>/dev/null | sort -h | tail -15'
+```
+
+`-x` keeps `/mnt/entropy` out automatically (separate mount). Then drill `/home` and `/var` with
+`sudo du -x -d2`. Typical 2026-10 shape: `/home` ≈ 1.4T (models 785G, `.local/share/containers` 334G,
+`radiance-state` 69G, corpora 64G, campaign 45G), `/var` ≈ 215G (docker 119G, plex 80G), `/opt` 17G.
+Verify live; sizes are disposable.
+
+Do NOT `grep -R` across `/home/dank/Containers` or `~/Projects` — jellyfin cache JSON makes it run
+for minutes. Scope `find -maxdepth 3 -iname '*harbor*'` and check unit dirs only.
+
+## Prove each target unreferenced before deleting
+
+- `sudo lsof <paths>` (fuse warnings are noise) and, for podman image IDs, cross-check
+  `podman ps -a --format '{{.ImageID}}'`; `podman image prune` keeps an in-use untagged image, but
+  confirm the container survives.
+- Rootless vs rootful matters: `frank-archon` runs as rootless podman under `dank` even though its
+  unit is a root unit. Check both `sudo podman ps -a` and
+  `XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps -a`, including stopped containers, and inspect their
+  mounts (`podman inspect -f '{{range .Mounts}}{{.Source}} {{end}}'`).
+- `grep -RIl '<path>' /etc/systemd/system /home/dank/.config/systemd` must be empty.
+- A running archon mounts only `/models`, `radiance-state/tune` and the template — **not**
+  `radiance-state/kvcache`, so stale `blocks.bin`/`ckpts.bin` are genuinely orphaned.
+
+## Reclaim ladder (safest first)
+
+1. **Orphan radiance KV spill** — `radiance-state/kvcache/frank-archon/{blocks.bin,ckpts.bin}`
+   (measured 69G, last write weeks old, no handle). `sudo rm -f` both. This alone can clear the
+   campaign gate.
+2. **Docker** — only if `systemctl is-enabled docker` is `disabled` and `docker ps -aq` is empty:
+   `sudo docker system prune -a -f` (measured ~104GB; volumes are NOT removed, keep it that way).
+   Verify `docker volume ls` still lists the volumes.
+3. **Podman dangling only** — `podman image prune -f` (default dangling, no `-a` without approval).
+   Measured 33 removed / ~62G freed; one dangling ID is the running calibration image and is skipped.
+   The remote shell is **zsh**: unquoted `$ids` does not word-split, so `podman rmi $ids` receives a
+   newline-joined string and fails with `invalid reference format`. Pipe IDs through xargs instead:
+   `podman images --format '{{.ID}} {{.Repository}}' | grep goharbor | cut -d' ' -f1 | sort -u | xargs -r podman rmi`.
+4. **Harbor registry images** — all `goharbor/*` v2.13 images are ~2.1G total and safe once no
+   harbor container/unit/compose exists (`~/.harbor` is a cmdh manifest, not the goharbor stack). The
+   big `harbor-hf`/`harbor-vllm` build images live under Docker and disappear with step 2.
+5. **Plex DB backups** — in `Plug-in Support/Databases`, keep the newest `*.db-YYYY-MM-DD` set and
+   delete older sets (measured 3 × 4.9G blobs + 3 × 1.4G library ≈ 19G). Plex stays up; never touch
+   the live `library.db`/`blobs.db`.
+6. **Later, with separate approval** — Swift-era `corpora/calib` + `corpora/kld-ref` (KLD header
+   names the base: `model` field), `uv cache prune` (~25G), `/opt/vllm`, `/opt/llama.cpp`,
+   `~/.local/bin` stale llama binaries, journal vacuum, old snaps. Tagged-but-unused podman builds
+   (vllm/rocm/fp030) remain ~225G; removing them loses build caches, so scope them explicitly.
+
+## The campaign watcher is the side effect
+
+`campaign/qwen38-w5nl64/window-when-space.sh` polls
+`df --output=avail -B 1M <campaign>/calib` every 60 s for `Q38_CALIB_MIN_FREE_MIB` (default 61440).
+The moment reclaim opens the gate it starts the documented calibration window, which **stops
+frank-archon** and boots `qwen38-bf16` on port 7770. Expect production downtime and a `health=000`
+during load; the engine is healthy once `/health` returns 200. Do not restart frank-archon by hand
+during the window — it would fight the rig for both cards.
+
+If the window fails, read `logs/window-start-*.log` and `/home/dank/logs/qwen38_bf16_calib.log`:
+`I stopping: draining in-flight requests` + 429 `the server is shutting down` means the orchestrator
+SIGTERMed a healthy engine (tap marker state), not a crash; a retry may then fail with
+`E startup stopped: device error` while the prior process's `VM memory stats ... non-zero when fini`
+clears. Restore is automatic: the window script restores frank-archon and reports the banked/kept
+`.rad`.
+
+## Verify
+
+```sh
+df -h /; df --output=avail -B 1M /
+XDG_RUNTIME_DIR=/run/user/$(id -u) podman system df
+curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
+XDG_RUNTIME_DIR=/run/user/$(id -u) podman ps --format '{{.Names}} {{.Status}}'
+```
+
+All running containers must be untouched, archon back to 200, and the campaign gate recomputed with
+its own command. ext4 reserves 5% (~90G on the 1.8T root): `df` Avail is what the unprivileged
+campaign sees, so root's ability to write does not mean the gate passes; lowering the reserve
+(`tune2fs -m 1`) releases ~72G but removes the fragmentation cushion on a nearly full disk — last
+resort only.