--- name: fleet-llm-payload-purge-refresh description: "Safely purge and refresh LLM model payloads on the Dank fleet (Dankclaw/Dankserver): protect live service payloads, confirm upstream re-fetchability, fingerprint before deleting, and use each service's own refresh path (halogen's first-start HALOGEN_DOWNLOAD vs frank-small's plain-GGUF fetch) — including the empty-bind-mount-dir trap." --- # Purge and refresh LLM model payloads (Dank fleet) Use when asked to clear / reclaim / refresh model caches on a host that is *serving* users, or to prove that a payload is up to date. ## 1. Inventory before touching anything ```bash du -sh /home/dank/models/* | sort -h # what is actually there ``` Then find which of those the live services mount — never delete a payload a unit mounts without an explicit, in-this-conversation instruction: ```bash grep -aE "^(Volume|Environment=.*(MODEL|CHECKPOINT|MODELS))" \ /home/dank/.config/containers/systemd/*.container \ /etc/systemd/system/frank-*.service ``` On Dankclaw the protected set has been: `models/halogen` (frank-large's .hgn checkpoint dir), `models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp` (frank-small's GGUF + mmproj). ComfyUI uses `/home/dank/comfyui-data/models`, **not** `models/checkpoints` — verify, because `models/checkpoints` looks like cache but has held Flux weights whose upstream is licence-gated. ## 2. Confirm each payload is re-fetchable BEFORE deleting it Query the Hub anonymously; a gated repo 403s even with the host token: ```bash curl -s "https://huggingface.co/api/models//tree/main?recursive=true&blobs=true" \ | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]" ``` An error/empty result means it is NOT safe to delete. Also filter by **quant name**, never by `endswith('.gguf')` — mradermacher ships large quants as `.partNofM` fragments which that test hides. ## 3. Fingerprint, then delete Record byte sizes (and sha256 when the file matters) **before** removal: that is the only way to answer "did the refresh actually change anything" afterwards. Per this repo's policy `rm` needs explicit operator permission; otherwise stage into `~/.AGENTTRASH/-/`. Note that a same-filesystem trash reclaims nothing — say so rather than implying space was freed. Root-owned leftovers (a trashed swapfile) need `sudo`, and check `swapon --show` first. ## 4. Refresh, per service **frank-large (halogen)** — the engine fetches for you. Clear the directory but leave it **existing and empty**, then start the unit: ```bash rm -rf /home/dank/models/halogen/* # keep the directory itself systemctl --user start frank-large.service journalctl --user -u frank-large.service -f | grep -a halogen: ``` The engine sees the checkpoint absent and runs `HALOGEN_DOWNLOAD` (`peonist-ai/halogen-qwen3.8-flash-next`) into the mount via `huggingface_hub`, resuming if interrupted. **Trap:** podman refuses a bind mount whose source directory is missing — `Error: statfs /home/dank/models/halogen: no such file or directory`, with the unit looping on `Restart=on-failure`. `mkdir -p` the empty directory and restart. ~124 GiB at 130-240 MB/s ≈ 10-20 min, then load. **frank-small (llama.cpp)** — no self-fetch; the serve script refuses to start with weights missing, so pull them yourself and verify: ```bash DEST=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp # repo ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF # files: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf + mmproj-Qwen3.8-27B-BF16.gguf curl -fL -C - --retry 10 -o "$DEST/" \ "https://huggingface.co//resolve/main/" # -f is mandatory sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to the LFS oid + GGUF magic ``` `curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — always `-f`, always compare byte counts to the Hub's declared size, always check the LFS sha256. Public repos need no token: omit `-H "Authorization: Bearer ..."` from argv entirely (it is world-readable in `/proc//cmdline`). ## 5. Verify, then record ```bash curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health # frank-large curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7771/health # frank-small curl -s http://127.0.0.1:7770/v1/models; curl -s http://127.0.0.1:7771/v1/models # a REAL completion on both, then halogen's boot ledger: journalctl --user -u frank-large.service | grep -aE "weights locked|kv pool:|left for everything" ``` Also sweep the vhosts (`www`, `grafana`, `prometheus`, `hermes`, `marreta`, `gitea`, `loki` — `curl -sk --resolve :443:192.168.0.143`) because a purge touches state the whole host shares. ## 6. Things that bite - **Sidecars are versioned in place.** Equal file sizes are not proof the repo is unchanged, and a stale local sidecar is invisible from its name. Compare local sizes against the Hub's declared sizes; that is how a 2.31 GiB → 2.40 GiB overlay update was caught. - **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark.** The A* run commits ~116 GiB of host memory and leaves ~10 GiB available; adding cache churn there risks the OOM/state failures this fleet has already seen. - **Reboot is the repair of last resort** if rootless podman's pause process dies during the work (`invalid internal status … could not find any running process`); `podman system migrate` panics in 5.7.0. - Because a peer session may be committing to `serverAppConfig` concurrently, re-read a doc region immediately before editing it, and check that the edit response's neighbouring lines are the ones you expect.