Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
210 lines
9.6 KiB
Markdown
210 lines
9.6 KiB
Markdown
---
|
||
type: reference
|
||
title: Payload and Disk Reclaim
|
||
description: "Disk pressure procedure: measure every bucket, classify, prove nothing references the payload, stage, purge on approval."
|
||
tags: [operations, disk, podman, payloads, huggingface, trash]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# Payload and Disk Reclaim
|
||
|
||
Standing procedure for "clean up old models" / "disk is full" on a
|
||
serverAppConfigs host. Order matters: **measure, prove nothing live uses it,
|
||
stage, ask.** Staging frees nothing until the user approves the purge — say so
|
||
up front or the request reads as unfulfilled.
|
||
|
||
## 1. Measure the whole filesystem, not the category asked about
|
||
|
||
```sh
|
||
du -sh /home/dank/models /home/dank/.local /home/dank/.cache \
|
||
/home/dank/comfyui-data /home/dank/GitPackages 2>/dev/null
|
||
df -h /home/dank | tail -1
|
||
podman system df # reclaimable image space
|
||
du -sh ~/.local/share/containers # where rootless storage physically is
|
||
```
|
||
|
||
The answer is usually NOT the category the user named. On Dankclaw
|
||
(2026-09-28) retired models were 510G, but podman images were **279G
|
||
reclaimable (95%)** and uv+pip caches **73G** — 352G of regenerable garbage,
|
||
bigger than the models. Lead with that finding; it changes the decision.
|
||
|
||
Check for sparse files before claiming a saving:
|
||
|
||
```sh
|
||
stat -c 'apparent=%s blocks=%b' <file>; du -sh --apparent-size <file>; du -sh <file>
|
||
```
|
||
|
||
A 3.3G "log" (`logs/x4e-run.log`) was **4K real** (8 blocks × 512) — deleting
|
||
it frees nothing. `du` vs `du --apparent-size` finds these instantly.
|
||
|
||
## 2. Classify against the runbook index
|
||
|
||
Every payload directory maps to a runtime in the AGENTS.md retired-runbook
|
||
table. Three buckets:
|
||
|
||
- **live** — referenced by a running service's env/unit.
|
||
- **rollback** — named as a rollback target in a runbook. ASK before
|
||
touching: they are the difference between a one-minute flip and a 30G
|
||
re-download.
|
||
- **retired** — maps to a retired runtime. Safe candidates.
|
||
|
||
Trap: a directory can hold **both** a dead target and a live asset — an era's
|
||
drafter can sit beside the retired target it came with (per-endpoint era tables:
|
||
[[frank-dragoon]], [[frank-zealot]], [[frank-archon]]). Move the
|
||
single file, never the directory, and say so in the manifest.
|
||
|
||
## 3. The live-reference check — before moving anything
|
||
|
||
```sh
|
||
for d in <candidate dirs>; do
|
||
grep -rl "$d" --include="*.sh" --include="*.env" --include="*.service" \
|
||
--include="*.container" --include="*.yaml" . | grep -v '^./old/'
|
||
done
|
||
```
|
||
|
||
Then **read every hit**; do not count them. Most are benign (comments,
|
||
retired fetch-script destinations, `retire_*_quadlet` helpers that move a
|
||
quadlet not a model) — but on Dankclaw this check found a live bug:
|
||
`serve-frank-zealot.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
|
||
payload retired a week earlier. The env file always overrode it, so it was
|
||
invisible; a missing env file would have produced a "missing asset" naming a
|
||
model that no longer exists.
|
||
|
||
**Durable lesson:** `:`-defaults in a serve script are never exercised in
|
||
production, so they rot silently. When reclaiming a payload, grep serve
|
||
scripts too — the fallback is a live reference in all but name.
|
||
|
||
Also check service **env files** (`scripts/*.env`) for `MODEL_DIR`/`MODEL`
|
||
paths, not just unit `Volume=` lines: a payload can be mounted by path and
|
||
named by env. `models/logs` and small tool dirs are not cache — logs are
|
||
evidence, keep them.
|
||
|
||
## 4. Stage, never delete
|
||
|
||
```sh
|
||
STAGE=/home/dank/.AGENTTRASH/$(date +%Y%m%d)-<topic>
|
||
mkdir -p "$STAGE"
|
||
mv <path> "$STAGE"/
|
||
```
|
||
|
||
Same filesystem ⇒ instant and recoverable — **and reclaims no space**; say so
|
||
rather than implying space was freed. Write `MANIFEST.md` into the stage dir:
|
||
per item — path, size, owning runtime, when retired, the `mv` command to
|
||
restore. Record what you **deliberately kept** and why; that is what a future
|
||
reader needs. Always check for pre-existing staged trash
|
||
(`du -sh ~/.AGENTTRASH/*`): 118G from a 2026-09-21 cleanup sat un-purged;
|
||
report the running total, not just the new batch. Root-owned leftovers need
|
||
`sudo`, and check `swapon --show` first (a trashed swapfile must not be
|
||
active).
|
||
|
||
## 5. Verify, then report — and do not purge
|
||
|
||
```sh
|
||
systemctl is-active frank-dragoon frank-zealot
|
||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
|
||
```
|
||
|
||
Report: staged items and sizes, what was kept and why, what is **still not
|
||
freed**, remaining candidates untouched. Purge only on explicit approval.
|
||
|
||
## Repo-flow gotcha: do NOT push before you deploy
|
||
|
||
`scripts/deploy.sh` sets `BASE_REVISION=$(git rev-parse origin/main)`. If you
|
||
pushed separately, the diff is empty and the planner reports "changed
|
||
services: (none)" — looks like a planner bug, is not one. The tool is
|
||
designed for the **unpushed** state: edit → test → `deploy.sh` (it pushes,
|
||
then dispatches). If you already pushed, pull on the host instead, and reason
|
||
about whether a restart is actually owed. Any serve-script/env change needs
|
||
`./tests/run.sh` before commit; a `:`-default change is usually **inert**
|
||
(env overrides it) and owes no restart — but say so explicitly.
|
||
|
||
# Purge and refresh protocol
|
||
|
||
When the task is clearing/reclaiming/**refreshing** model caches on a serving
|
||
host, add these steps.
|
||
|
||
## Fingerprint before deleting
|
||
|
||
`ls -l --block-size=1 <dir>`; for LLM payloads compare each file against the
|
||
Hub's declared size. Fingerprints are the only way to answer "did the refresh
|
||
change anything" afterwards — **a refresh is also an update check**: if
|
||
upstream never changed, the bytes come back identical, which is itself the
|
||
finding. Sidecars are versioned **in place**: equal main-checkpoint sizes do
|
||
not prove the repo is unchanged, and a stale sidecar is invisible from its
|
||
name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560 while the main
|
||
checkpoint returned byte-identical). If a file is irreplaceable upstream
|
||
(delisted revision, gated repo), say so before deleting.
|
||
|
||
## Confirm re-fetchability first
|
||
|
||
Query the Hub anonymously; a gated repo 403s even with the host's token:
|
||
|
||
```bash
|
||
curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
|
||
| python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"
|
||
```
|
||
|
||
An error/empty result means it is NOT safe to delete. Filter by **quant
|
||
name**, never `endswith('.gguf')` — mradermacher ships large quants as
|
||
`.partNofM` fragments that filter hides.
|
||
|
||
## Refresh per engine — know the recovery path BEFORE deleting
|
||
|
||
- **frank-dragoon**: self-fetching — clear the directory but
|
||
leave it **existing and empty**, then start the unit; the download knobs and
|
||
the payload's byte count (which is what the re-download costs) are on
|
||
[[frank-dragoon]]. **Bind-mount trap:**
|
||
podman refuses a mount whose source is missing
|
||
(`Error: statfs /home/dank/models/halogen: no such file or directory`) and
|
||
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir.
|
||
- **llama.cpp services**: no self-fetch; the serve script refuses to start
|
||
without weights. Pull yourself and verify:
|
||
|
||
```bash
|
||
curl -fL -C - --retry 10 -o "$DEST/<file>" \
|
||
"https://huggingface.co/<repo>/resolve/main/<file>" # -f is mandatory
|
||
sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to LFS oid + GGUF magic
|
||
```
|
||
|
||
`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — a
|
||
gated repo once produced three 153-byte shards with identical sha256s and a
|
||
cheerful DONE line for 104 GiB. Always `-fL -C -`, compare byte counts to the
|
||
Hub's declared sizes, compare sha256 to the LFS oid. Assemble `.partNofM`
|
||
fragments with `cat part1 part2 part3 > name.gguf` in numeric order (per-part
|
||
hashes verify the whole — concatenation is deterministic). Public repos need
|
||
no token: omit `-H "Authorization: Bearer …"` from argv entirely (it is
|
||
world-readable in `/proc/<pid>/cmdline`).
|
||
|
||
## Restore, then verify behaviour — not health alone
|
||
|
||
Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`),
|
||
`reset-failed`, `start` (root units need `sudo`). Watch the engine's own
|
||
fetch/load lines, then `/health`, `/v1/models`, and a **real completion** on
|
||
every restored endpoint. Read the engine's memory ledger (kv pool / weights /
|
||
blocks left) rather than assuming the config still fits. Sweep the vhosts too
|
||
(`www`, `grafana`, `prometheus`, `hermes`, `marreta`, `gitea`, `loki` via
|
||
`curl -sk --resolve <host>:443:192.168.0.143`) — a purge touches state the
|
||
whole host shares.
|
||
|
||
## Things that bite
|
||
|
||
- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark** —
|
||
the A* run commits ~116 GiB and leaves ~10 GiB available; cache churn risks
|
||
the OOM/state failures this fleet has already seen.
|
||
- **Reboot is the repair of last resort** if rootless podman's pause process
|
||
dies mid-work (`invalid internal status … could not find any running
|
||
process`); `podman system migrate` panics in 5.7.0.
|
||
- A same-filesystem trash reclaims no space; root-owned leftovers need
|
||
`sudo`; never `rm` without explicit permission.
|
||
- **A giant image layer is not automatically a model blob.** The 2026-10-17 census found a 121 GB layer
|
||
(`16b6663d…`) that read like an undeleted payload; it is the **engine image** — the radiance runtime itself.
|
||
Map each layer to what the serving trio actually mounts before deleting anything: that census is both the
|
||
disk-reclaim proof and the check that no deleted payload path still backs a live container.
|
||
|
||
## Related
|
||
|
||
- [[Model Swap and Rollback]] — the runbook index this classification reads
|
||
- [[Quadlet Lifecycle]] — restoring the unit after a payload refresh
|
||
- [[Remote Job Runner]] — supervising long re-fetches
|
||
- [[Deploy State Triage]] — the push-before-deploy trap in context
|