serverAppConfigs/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Payload and Disk Reclaim	Disk pressure procedure: measure every bucket, classify, prove nothing references the payload, stage, purge on approval.	operations disk podman payloads huggingface trash	2026-10-05T00:00:00-07:00

Payload and Disk Reclaim

Standing procedure for "clean up old models" / "disk is full" on a serverAppConfigs host. Order matters: measure, prove nothing live uses it, stage, ask. Staging frees nothing until the user approves the purge — say so up front or the request reads as unfulfilled.

1. Measure the whole filesystem, not the category asked about

du -sh /home/dank/models /home/dank/.local /home/dank/.cache \
       /home/dank/comfyui-data /home/dank/GitPackages 2>/dev/null
df -h /home/dank | tail -1
podman system df          # reclaimable image space
du -sh ~/.local/share/containers   # where rootless storage physically is

The answer is usually NOT the category the user named. On Dankclaw (2026-09-28) retired models were 510G, but podman images were 279G reclaimable (95%) and uv+pip caches 73G — 352G of regenerable garbage, bigger than the models. Lead with that finding; it changes the decision.

Check for sparse files before claiming a saving:

stat -c 'apparent=%s blocks=%b' <file>; du -sh --apparent-size <file>; du -sh <file>

A 3.3G "log" (logs/x4e-run.log) was 4K real (8 blocks × 512) — deleting it frees nothing. du vs du --apparent-size finds these instantly.

2. Classify against the runbook index

Every payload directory maps to a runtime in the AGENTS.md retired-runbook table. Three buckets:

• live — referenced by a running service's env/unit.
• rollback — named as a rollback target in a runbook. ASK before touching: they are the difference between a one-minute flip and a 30G re-download.
• retired — maps to a retired runtime. Safe candidates.

Trap: a directory can hold both a dead target and a live asset — an era's drafter can sit beside the retired target it came with (per-endpoint era tables: frank-dragoon, frank-zealot, frank-archon). Move the single file, never the directory, and say so in the manifest.

3. The live-reference check — before moving anything

for d in <candidate dirs>; do
  grep -rl "$d" --include="*.sh" --include="*.env" --include="*.service" \
            --include="*.container" --include="*.yaml" . | grep -v '^./old/'
done

Then read every hit; do not count them. Most are benign (comments, retired fetch-script destinations, retire_*_quadlet helpers that move a quadlet not a model) — but on Dankclaw this check found a live bug: serve-frank-zealot.sh's : "${MODEL_DIR:=...}" fallback still pointed at a payload retired a week earlier. The env file always overrode it, so it was invisible; a missing env file would have produced a "missing asset" naming a model that no longer exists.

Durable lesson: :-defaults in a serve script are never exercised in production, so they rot silently. When reclaiming a payload, grep serve scripts too — the fallback is a live reference in all but name.

Also check service env files (scripts/*.env) for MODEL_DIR/MODEL paths, not just unit Volume= lines: a payload can be mounted by path and named by env. models/logs and small tool dirs are not cache — logs are evidence, keep them.

4. Stage, never delete

STAGE=/home/dank/.AGENTTRASH/$(date +%Y%m%d)-<topic>
mkdir -p "$STAGE"
mv <path> "$STAGE"/

Same filesystem ⇒ instant and recoverable — and reclaims no space; say so rather than implying space was freed. Write MANIFEST.md into the stage dir: per item — path, size, owning runtime, when retired, the mv command to restore. Record what you deliberately kept and why; that is what a future reader needs. Always check for pre-existing staged trash (du -sh ~/.AGENTTRASH/*): 118G from a 2026-09-21 cleanup sat un-purged; report the running total, not just the new batch. Root-owned leftovers need sudo, and check swapon --show first (a trashed swapfile must not be active).

5. Verify, then report — and do not purge

systemctl is-active frank-dragoon frank-zealot
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health

Report: staged items and sizes, what was kept and why, what is still not freed, remaining candidates untouched. Purge only on explicit approval.

Repo-flow gotcha: do NOT push before you deploy

scripts/deploy.sh sets BASE_REVISION=$(git rev-parse origin/main). If you pushed separately, the diff is empty and the planner reports "changed services: (none)" — looks like a planner bug, is not one. The tool is designed for the unpushed state: edit → test → deploy.sh (it pushes, then dispatches). If you already pushed, pull on the host instead, and reason about whether a restart is actually owed. Any serve-script/env change needs ./tests/run.sh before commit; a :-default change is usually inert (env overrides it) and owes no restart — but say so explicitly.

Purge and refresh protocol

When the task is clearing/reclaiming/refreshing model caches on a serving host, add these steps.

Fingerprint before deleting

ls -l --block-size=1 <dir>; for LLM payloads compare each file against the Hub's declared size. Fingerprints are the only way to answer "did the refresh change anything" afterwards — a refresh is also an update check: if upstream never changed, the bytes come back identical, which is itself the finding. Sidecars are versioned in place: equal main-checkpoint sizes do not prove the repo is unchanged, and a stale sidecar is invisible from its name (observed: overlay.hgn 2,477,677,120 → 2,572,466,560 while the main checkpoint returned byte-identical). If a file is irreplaceable upstream (delisted revision, gated repo), say so before deleting.

Confirm re-fetchability first

Query the Hub anonymously; a gated repo 403s even with the host's token:

curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"

An error/empty result means it is NOT safe to delete. Filter by quant name, never endswith('.gguf') — mradermacher ships large quants as .partNofM fragments that filter hides.

Refresh per engine — know the recovery path BEFORE deleting

• frank-dragoon: self-fetching — clear the directory but leave it existing and empty, then start the unit; the download knobs and the payload's byte count (which is what the re-download costs) are on frank-dragoon. Bind-mount trap: podman refuses a mount whose source is missing (Error: statfs /home/dank/models/halogen: no such file or directory) and the unit loops on Restart=on-failure — mkdir -p the empty dir.
• llama.cpp services: no self-fetch; the serve script refuses to start without weights. Pull yourself and verify:

curl -fL -C - --retry 10 -o "$DEST/<file>" \
  "https://huggingface.co/<repo>/resolve/main/<file>"   # -f is mandatory
sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf   # compare to LFS oid + GGUF magic

curl exits 0 on HTTP 401/403 and writes the error body as the artifact — a gated repo once produced three 153-byte shards with identical sha256s and a cheerful DONE line for 104 GiB. Always -fL -C -, compare byte counts to the Hub's declared sizes, compare sha256 to the LFS oid. Assemble .partNofM fragments with cat part1 part2 part3 > name.gguf in numeric order (per-part hashes verify the whole — concatenation is deterministic). Public repos need no token: omit -H "Authorization: Bearer …" from argv entirely (it is world-readable in /proc/<pid>/cmdline).

Restore, then verify behaviour — not health alone

Put the quadlet back (mv from trash + systemctl --user daemon-reload) reset-failed, start (root units need sudo). Watch the engine's own fetch/load lines, then /health, /v1/models, and a real completion on every restored endpoint. Read the engine's memory ledger (kv pool / weights / blocks left) rather than assuming the config still fits. Sweep the vhosts too (www, grafana, prometheus, hermes, marreta, gitea, loki curl -sk --resolve <host>:443:192.168.0.143) — a purge touches state the whole host shares.

Things that bite

• Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark — the A* run commits ~116 GiB and leaves ~10 GiB available; cache churn risks the OOM/state failures this fleet has already seen.
• Reboot is the repair of last resort if rootless podman's pause process dies mid-work (invalid internal status … could not find any running process); podman system migrate panics in 5.7.0.
• A same-filesystem trash reclaims no space; root-owned leftovers need sudo; never rm without explicit permission.
• A giant image layer is not automatically a model blob. The 2026-10-17 census found a 121 GB layer (16b6663d…) that read like an undeleted payload; it is the engine image — the radiance runtime itself. Map each layer to what the serving trio actually mounts before deleting anything: that census is both the disk-reclaim proof and the check that no deleted payload path still backs a live container.

Related

• Model Swap and Rollback — the runbook index this classification reads
• Quadlet Lifecycle — restoring the unit after a payload refresh
• Remote Job Runner — supervising long re-fetches
• Deploy State Triage — the push-before-deploy trap in context