0 Fork 0

Files

T

Frank 33eae71d33 docs: distill the managed skill library into the vault (waves 1-7)

All ~173 managed skills read in full and merged, one polished reference
page per skill-cluster:

- LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving
  Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4
  Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot
  Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM
  Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops,
  Heterogeneous Split Triage, FreeToken Builds + deltas folded into the
  retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V
  qualification pipeline, FLM pack registration, ik hybrid envelope,
  frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch
  split section, bench-window D-state/stale-read notes, second-endpoint
  bring-up).
- Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State
  Triage.
- Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim,
  Container Host Reachability, Fleet Ops Playbook, Fleet Audit and
  Posture, SSH and Remote Access.
- Infrastructure: Observability Provisioning + Homepage Status Probes;
  updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun
  zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page
  preserved and extended).
- FrankBook-Pro: ARR-Stack filled (API split, schema-first probing,
  auth-posture, auth-config trap); qBittorrent E2E verification.
- Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and
  Edits, Command Center Dotfiles, PICO-8 Asset Standards.
- Indexes: LLM Cross-Cutting References section, Operations/ and Agent/
  index pages, root-index Skill Distillation section, AGENTS.md pointer.
- log.md carries the wave plan.

check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.

2026-10-05 21:19:15 -07:00

9.3 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Payload and Disk Reclaim

Standing procedure for "clean up old models" / "disk is full" on a serverAppConfigs host. Order matters: measure, prove nothing live uses it, stage, ask. Staging frees nothing until the user approves the purge — say so up front or the request reads as unfulfilled.

1. Measure the whole filesystem, not the category asked about

du -sh /home/dank/models /home/dank/.local /home/dank/.cache \
       /home/dank/comfyui-data /home/dank/GitPackages 2>/dev/null
df -h /home/dank | tail -1
podman system df          # reclaimable image space
du -sh ~/.local/share/containers   # where rootless storage physically is

The answer is usually NOT the category the user named. On Dankclaw (2026-09-28) retired models were 510G, but podman images were 279G reclaimable (95%) and uv+pip caches 73G — 352G of regenerable garbage, bigger than the models. Lead with that finding; it changes the decision.

Check for sparse files before claiming a saving:

stat -c 'apparent=%s blocks=%b' <file>; du -sh --apparent-size <file>; du -sh <file>

A 3.3G "log" (logs/x4e-run.log) was 4K real (8 blocks × 512) — deleting it frees nothing. du vs du --apparent-size finds these instantly.

2. Classify against the runbook index

Every payload directory maps to a runtime in the AGENTS.md retired-runbook table. Three buckets:

live — referenced by a running service's env/unit.
rollback — named as a rollback target in a runbook. ASK before touching: they are the difference between a one-minute flip and a 30G re-download.
retired — maps to a retired runtime. Safe candidates.

Trap: a directory can hold both a dead target and a live drafter (frank-large's DFlash2 drafter lives beside the previous target). Move the single file, never the directory, and say so in the manifest.

3. The live-reference check — before moving anything

for d in <candidate dirs>; do
  grep -rl "$d" --include="*.sh" --include="*.env" --include="*.service" \
            --include="*.container" --include="*.yaml" . | grep -v '^./old/'
done

Then read every hit; do not count them. Most are benign (comments, retired fetch-script destinations, retire_*_quadlet helpers that move a quadlet not a model) — but on Dankclaw this check found a live bug: serve-frank-small.sh's : "${MODEL_DIR:=...}" fallback still pointed at a payload retired a week earlier. The env file always overrode it, so it was invisible; a missing env file would have produced a "missing asset" naming a model that no longer exists.

Durable lesson: :-defaults in a serve script are never exercised in production, so they rot silently. When reclaiming a payload, grep serve scripts too — the fallback is a live reference in all but name.

Also check service env files (scripts/*.env) for MODEL_DIR/MODEL paths, not just unit Volume= lines: a payload can be mounted by path and named by env. models/logs and small tool dirs are not cache — logs are evidence, keep them.

4. Stage, never delete

STAGE=/home/dank/.AGENTTRASH/$(date +%Y%m%d)-<topic>
mkdir -p "$STAGE"
mv <path> "$STAGE"/

Same filesystem ⇒ instant and recoverable — and reclaims no space; say so rather than implying space was freed. Write MANIFEST.md into the stage dir: per item — path, size, owning runtime, when retired, the mv command to restore. Record what you deliberately kept and why; that is what a future reader needs. Always check for pre-existing staged trash (du -sh ~/.AGENTTRASH/*): 118G from a 2026-09-21 cleanup sat un-purged; report the running total, not just the new batch. Root-owned leftovers need sudo, and check swapon --show first (a trashed swapfile must not be active).

5. Verify, then report — and do not purge

systemctl is-active frank-large frank-small
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health

Report: staged items and sizes, what was kept and why, what is still not freed, remaining candidates untouched. Purge only on explicit approval.

Repo-flow gotcha: do NOT push before you deploy

scripts/deploy.sh sets BASE_REVISION=$(git rev-parse origin/main). If you pushed separately, the diff is empty and the planner reports "changed services: (none)" — looks like a planner bug, is not one. The tool is designed for the unpushed state: edit → test → deploy.sh (it pushes, then dispatches). If you already pushed, pull on the host instead, and reason about whether a restart is actually owed. Any serve-script/env change needs ./tests/run.sh before commit; a :-default change is usually inert (env overrides it) and owes no restart — but say so explicitly.

Purge and refresh protocol

When the task is clearing/reclaiming/refreshing model caches on a serving host, add these steps.

Fingerprint before deleting

ls -l --block-size=1 <dir>; for LLM payloads compare each file against the Hub's declared size. Fingerprints are the only way to answer "did the refresh change anything" afterwards — a refresh is also an update check: if upstream never changed, the bytes come back identical, which is itself the finding. Sidecars are versioned in place: equal main-checkpoint sizes do not prove the repo is unchanged, and a stale sidecar is invisible from its name (observed: overlay.hgn 2,477,677,120 → 2,572,466,560 while the main checkpoint returned byte-identical). If a file is irreplaceable upstream (delisted revision, gated repo), say so before deleting.

Confirm re-fetchability first

Query the Hub anonymously; a gated repo 403s even with the host's token:

curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"

An error/empty result means it is NOT safe to delete. Filter by quant name, never endswith('.gguf') — mradermacher ships large quants as .partNofM fragments that filter hides.

Refresh per engine — know the recovery path BEFORE deleting

frank-large (halogen): the engine fetches its own payload via HALOGEN_DOWNLOAD when the checkpoint is absent. Clear the directory but leave it existing and empty, then start the unit. Bind-mount trap: podman refuses a mount whose source is missing (Error: statfs /home/dank/models/halogen: no such file or directory) and the unit loops on Restart=on-failure — mkdir -p the empty dir. ~124 GiB at 130–240 MB/s ≈ 10–20 min, then load.
llama.cpp services: no self-fetch; the serve script refuses to start without weights. Pull yourself and verify:

curl -fL -C - --retry 10 -o "$DEST/<file>" \
  "https://huggingface.co/<repo>/resolve/main/<file>"   # -f is mandatory
sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf   # compare to LFS oid + GGUF magic

curl exits 0 on HTTP 401/403 and writes the error body as the artifact — a gated repo once produced three 153-byte shards with identical sha256s and a cheerful DONE line for 104 GiB. Always -fL -C -, compare byte counts to the Hub's declared sizes, compare sha256 to the LFS oid. Assemble .partNofM fragments with cat part1 part2 part3 > name.gguf in numeric order (per-part hashes verify the whole — concatenation is deterministic). Public repos need no token: omit -H "Authorization: Bearer …" from argv entirely (it is world-readable in /proc/<pid>/cmdline).

Restore, then verify behaviour — not health alone

Put the quadlet back (mv from trash + systemctl --user daemon-reload), reset-failed, start (root units need sudo). Watch the engine's own fetch/load lines, then /health, /v1/models, and a real completion on every restored endpoint. Read the engine's memory ledger (kv pool / weights / blocks left) rather than assuming the config still fits. Sweep the vhosts too (www, grafana, prometheus, hermes, marreta, gitea, loki via curl -sk --resolve <host>:443:192.168.0.143) — a purge touches state the whole host shares.

Things that bite

Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark — the A* run commits ~116 GiB and leaves ~10 GiB available; cache churn risks the OOM/state failures this fleet has already seen.
Reboot is the repair of last resort if rootless podman's pause process dies mid-work (invalid internal status … could not find any running process); podman system migrate panics in 5.7.0.
A same-filesystem trash reclaims no space; root-owned leftovers need sudo; never rm without explicit permission.

Model Swap and Rollback — the runbook index this classification reads
Quadlet Lifecycle — restoring the unit after a payload refresh
Remote Job Runner — supervising long re-fetches
Deploy State Triage — the push-before-deploy trap in context

9.3 KiB Raw Blame History Unescape Escape