agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills

- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh
- models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes
- niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0
- memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills
- managed-skills: 40+ new fleet runbook skills
- agent/skills: import built-in skill set
This commit is contained in:
Frank committed 2026-09-17 12:27:08 -07:00
1 parent 83c9c92b3f
commit 135bb2194d
103 files changed
+5333 -170

No files matched your search

@@ -0,0 +1,117 @@
---
name: fleet-llm-payload-purge-refresh
description: "Safely purge and refresh LLM model payloads on the Dank fleet (Dankclaw/Dankserver): protect live service payloads, confirm upstream re-fetchability, fingerprint before deleting, and use each service's own refresh path (halogen's first-start HALOGEN_DOWNLOAD vs frank-small's plain-GGUF fetch) — including the empty-bind-mount-dir trap."
---
# Purge and refresh LLM model payloads (Dank fleet)
Use when asked to clear / reclaim / refresh model caches on a host that is
*serving* users, or to prove that a payload is up to date.
## 1. Inventory before touching anything
```bash
du -sh /home/dank/models/* | sort -h # what is actually there
```
Then find which of those the live services mount — never delete a payload a unit
mounts without an explicit, in-this-conversation instruction:
```bash
grep -aE "^(Volume|Environment=.*(MODEL|CHECKPOINT|MODELS))" \
/home/dank/.config/containers/systemd/*.container \
/etc/systemd/system/frank-*.service
```
On Dankclaw the protected set has been: `models/halogen` (frank-large's .hgn
checkpoint dir), `models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp` (frank-small's GGUF +
mmproj). ComfyUI uses `/home/dank/comfyui-data/models`, **not**
`models/checkpoints` — verify, because `models/checkpoints` looks like cache but
has held Flux weights whose upstream is licence-gated.
## 2. Confirm each payload is re-fetchable BEFORE deleting it
Query the Hub anonymously; a gated repo 403s even with the host token:
```bash
curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
| python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"
```
An error/empty result means it is NOT safe to delete. Also filter by **quant
name**, never by `endswith('.gguf')` — mradermacher ships large quants as
`.partNofM` fragments which that test hides.
## 3. Fingerprint, then delete
Record byte sizes (and sha256 when the file matters) **before** removal: that is
the only way to answer "did the refresh actually change anything" afterwards.
Per this repo's policy `rm` needs explicit operator permission; otherwise stage
into `~/.AGENTTRASH/<date>-<topic>/`. Note that a same-filesystem trash reclaims
nothing — say so rather than implying space was freed. Root-owned leftovers (a
trashed swapfile) need `sudo`, and check `swapon --show` first.
## 4. Refresh, per service
**frank-large (halogen)** — the engine fetches for you. Clear the directory but
leave it **existing and empty**, then start the unit:
```bash
rm -rf /home/dank/models/halogen/* # keep the directory itself
systemctl --user start frank-large.service
journalctl --user -u frank-large.service -f | grep -a halogen:
```
The engine sees the checkpoint absent and runs `HALOGEN_DOWNLOAD`
(`peonist-ai/halogen-qwen3.8-flash-next`) into the mount via `huggingface_hub`,
resuming if interrupted. **Trap:** podman refuses a bind mount whose source
directory is missing — `Error: statfs /home/dank/models/halogen: no such file or
directory`, with the unit looping on `Restart=on-failure`. `mkdir -p` the empty
directory and restart. ~124 GiB at 130-240 MB/s ≈ 10-20 min, then load.
**frank-small (llama.cpp)** — no self-fetch; the serve script refuses to start
with weights missing, so pull them yourself and verify:
```bash
DEST=/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp # repo ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
# files: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf + mmproj-Qwen3.8-27B-BF16.gguf
curl -fL -C - --retry 10 -o "$DEST/<file>" \
"https://huggingface.co/<repo>/resolve/main/<file>" # -f is mandatory
sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to the LFS oid + GGUF magic
```
`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — always
`-f`, always compare byte counts to the Hub's declared size, always check the LFS
sha256. Public repos need no token: omit `-H "Authorization: Bearer ..."` from
argv entirely (it is world-readable in `/proc/<pid>/cmdline`).
## 5. Verify, then record
```bash
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health # frank-large
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7771/health # frank-small
curl -s http://127.0.0.1:7770/v1/models; curl -s http://127.0.0.1:7771/v1/models
# a REAL completion on both, then halogen's boot ledger:
journalctl --user -u frank-large.service | grep -aE "weights locked|kv pool:|left for everything"
```
Also sweep the vhosts (`www`, `grafana`, `prometheus`, `hermes`, `marreta`,
`gitea`, `loki` — `curl -sk --resolve <host>:443:192.168.0.143`) because a purge
touches state the whole host shares.
## 6. Things that bite
- **Sidecars are versioned in place.** Equal file sizes are not proof the repo is
unchanged, and a stale local sidecar is invisible from its name. Compare local
sizes against the Hub's declared sizes; that is how a 2.31 GiB → 2.40 GiB
overlay update was caught.
- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark.** The
A* run commits ~116 GiB of host memory and leaves ~10 GiB available; adding
cache churn there risks the OOM/state failures this fleet has already seen.
- **Reboot is the repair of last resort** if rootless podman's pause process dies
during the work (`invalid internal status … could not find any running
process`); `podman system migrate` panics in 5.7.0.
- Because a peer session may be committing to `serverAppConfig` concurrently,
re-read a doc region immediately before editing it, and check that the edit
response's neighbouring lines are the ones you expect.