---
name: fleet-model-cache-purge-refresh
description: "Clear and re-fetch model payloads on a Dank-fleet host that serves production LLM endpoints — inventory, protect live payloads, verify every fetch, restore the service, and use the refresh to detect upstream updates."
---

Clearing a model cache on a serving host is a two-sided operation: the operator wants no stale
payload, and the host must keep serving. Do it in this order, on the target host.

## 1. Inventory before touching anything

```bash
du -sh /home/dank/models/* | sort -h      # payload dirs, biggest last
grep -rlE "models/|<model-name>" /home/dank/.config/containers/systemd/ /etc/systemd/system/*.service
```

Classify every dir: **live service payload** (referenced by a unit/env), **retired runtime
payload**, **unreferenced model dir**, **tool/script dir**, **logs** (evidence — keep). Check the
service env files (`scripts/*.env`) for `MODEL_DIR`/`MODEL` paths, not just unit `Volume=` lines:
a payload can be mounted by path and named by env. `models/logs` and small tool dirs are not cache.

## 2. Record fingerprints first

`ls -l --block-size=1 <dir>` gives sizes; for the LLM payloads compare each to the Hub's declared
size (`/api/models/<repo>/tree/main?recursive=true&blobs=true`). They can differ — see §6. If a
file is irreplaceable upstream (delisted revision, gated repo) say so before deleting it.

## 3. Protect live payloads by delete-then-refetch, never delete-only

Deleting a live payload leaves the endpoint unable to start. Know the recovery path *before*
deleting: some engines fetch their own weights (halogen: `HALOGEN_DOWNLOAD`, triggered when the
checkpoint is absent), others need a manual fetch (llama.cpp GGUF services: the serve script
refuses to start without the weights).

**Bind-mount trap:** podman refuses a mount whose source directory is missing —
`Error: statfs /home/dank/models/<dir>: no such file or directory`, with the unit looping on
`Restart=on-failure`. Recreate the directory **empty** (`mkdir -p`), and the engine's first-start
fetch runs.

## 4. Fetch with verification (never trust an exit code)

`curl -o FILE URL` exits 0 on 401/403 and writes the error body — a gated repo produced three
153-byte shards with identical sha256s and a cheerful DONE line. Always: `-fL -C -`, compare the
byte count against the Hub's declared size, and compare `sha256sum` against the LFS oid. Public
repos means **no token in argv** (it is world-readable in `/proc/<pid>/cmdline`).

**Parts trap:** mradermacher-style repos ship large quants as `.part1ofN … .partNofN` — a filter on
`endswith('.gguf')` makes those quants look absent. Fetch the parts, verify each, then
`cat part1 part2 part3 > name.gguf` in numeric order; the assembled file's size equals the sum
(concatenation is deterministic, so per-part hashes verify the whole).

## 5. Restore, then verify behaviour — not health alone

Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`), `reset-failed`, `start`;
root units need `sudo`. Watch the engine's own fetch/load lines in the journal, then check
`/health`, `/v1/models`, and a **real completion** on every restored endpoint. Read the engine's
memory ledger (kv pool / weights / blocks left) rather than assuming the config still fits.

## 6. A refresh is also an update check

Clearing and re-fetching only guarantees freshness *at fetch time* — if upstream has not changed,
the bytes come back identical, which is itself the finding. Compare recorded fingerprints against
the new local sizes and the Hub's declared sizes: sidecars are often versioned **in place**, so a
stale copy is invisible from its file name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560
while the main checkpoint returned byte-identical).

## 7. Close out

- Clear test payloads once the benchmark using them is done, and keep the payload the *next*
  experiment needs.
- Leave the operator's decision points explicit: unreferenced-but-licensed dirs, other services'
  caches, and whether they want the payloads absent at their next run or fresh as of now.
- Record sizes/verdicts in the vault (runbook + `log.md`); never `rm` without explicit permission,
  and remember a same-filesystem trash reclaims no space.