command-center/agent/managed-skills/fleet-llm-payload-purge-refresh/SKILL.md

name, description

name	description
fleet-llm-payload-purge-refresh	Safely purge and refresh LLM model payloads on the Dank fleet (Dankclaw/Dankserver): protect live service payloads, confirm upstream re-fetchability, fingerprint before deleting, and use each service's own refresh path (halogen's first-start HALOGEN_DOWNLOAD vs frank-small's plain-GGUF fetch) — including the empty-bind-mount-dir trap.

Purge and refresh LLM model payloads (Dank fleet)

Use when asked to clear / reclaim / refresh model caches on a host that is serving users, or to prove that a payload is up to date.

1. Inventory before touching anything

du -sh /home/dank/models/* | sort -h        # what is actually there

Then find which of those the live services mount — never delete a payload a unit mounts without an explicit, in-this-conversation instruction:

grep -aE "^(Volume|Environment=.*(MODEL|CHECKPOINT|MODELS))" \
  /home/dank/.config/containers/systemd/*.container \
  /etc/systemd/system/frank-*.service

On Dankclaw the protected set has been: models/halogen (frank-large's .hgn checkpoint dir), models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp (frank-small's GGUF + mmproj). ComfyUI uses /home/dank/comfyui-data/models, not models/checkpoints — verify, because models/checkpoints looks like cache but has held Flux weights whose upstream is licence-gated.

2. Confirm each payload is re-fetchable BEFORE deleting it

Query the Hub anonymously; a gated repo 403s even with the host token:

curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"

An error/empty result means it is NOT safe to delete. Also filter by quant name, never by endswith('.gguf') — mradermacher ships large quants as .partNofM fragments which that test hides.

3. Fingerprint, then delete

Record byte sizes (and sha256 when the file matters) before removal: that is the only way to answer "did the refresh actually change anything" afterwards.

Per this repo's policy rm needs explicit operator permission; otherwise stage into ~/.AGENTTRASH/<date>-<topic>/. Note that a same-filesystem trash reclaims nothing — say so rather than implying space was freed. Root-owned leftovers (a trashed swapfile) need sudo, and check swapon --show first.

4. Refresh, per service

frank-large (halogen) — the engine fetches for you. Clear the directory but leave it existing and empty, then start the unit:

rm -rf /home/dank/models/halogen/*        # keep the directory itself
systemctl --user start frank-large.service
journalctl --user -u frank-large.service -f | grep -a halogen:

The engine sees the checkpoint absent and runs HALOGEN_DOWNLOAD (peonist-ai/halogen-qwen3.8-flash-next) into the mount via huggingface_hub, resuming if interrupted. Trap: podman refuses a bind mount whose source directory is missing — Error: statfs /home/dank/models/halogen: no such file or directory, with the unit looping on Restart=on-failure. mkdir -p the empty directory and restart. ~124 GiB at 130-240 MB/s ≈ 10-20 min, then load.

frank-small (llama.cpp) — no self-fetch; the serve script refuses to start with weights missing, so pull them yourself and verify:

DEST=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp   # repo ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
# files: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf + mmproj-Qwen3.8-27B-BF16.gguf
curl -fL -C - --retry 10 -o "$DEST/<file>" \
  "https://huggingface.co/<repo>/resolve/main/<file>"   # -f is mandatory
sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf           # compare to the LFS oid + GGUF magic

curl exits 0 on HTTP 401/403 and writes the error body as the artifact — always -f, always compare byte counts to the Hub's declared size, always check the LFS sha256. Public repos need no token: omit -H "Authorization: Bearer ..." from argv entirely (it is world-readable in /proc/<pid>/cmdline).

5. Verify, then record

curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health   # frank-large
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7771/health   # frank-small
curl -s http://127.0.0.1:7770/v1/models; curl -s http://127.0.0.1:7771/v1/models
# a REAL completion on both, then halogen's boot ledger:
journalctl --user -u frank-large.service | grep -aE "weights locked|kv pool:|left for everything"

Also sweep the vhosts (www, grafana, prometheus, hermes, marreta, gitea, loki — curl -sk --resolve <host>:443:192.168.0.143) because a purge touches state the whole host shares.

6. Things that bite

• Sidecars are versioned in place. Equal file sizes are not proof the repo is unchanged, and a stale local sidecar is invisible from its name. Compare local sizes against the Hub's declared sizes; that is how a 2.31 GiB → 2.40 GiB overlay update was caught.
• Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark. The A* run commits ~116 GiB of host memory and leaves ~10 GiB available; adding cache churn there risks the OOM/state failures this fleet has already seen.
• Reboot is the repair of last resort if rootless podman's pause process dies during the work (invalid internal status … could not find any running process); podman system migrate panics in 5.7.0.
• Because a peer session may be committing to serverAppConfig concurrently, re-read a doc region immediately before editing it, and check that the edit response's neighbouring lines are the ones you expect.