Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
2.2 KiB
2.2 KiB
name, description
| name | description |
|---|---|
| llm-model-swap-fit-preflight | Preflight an LLM model swap for hardware fit before deploying; catch the silent swap-thrash outage mode (large RAM OOM, self-restarting + boot-enabled units) and recover. |
LLM Model-Swap Fit Preflight
Never deploy a model swap without confirming the checkpoint FITS, and know the failure mode when it doesn't: it is a silent, self-sustaining thrash, not a crash.
Preflight (before deploy)
- Get the checkpoint's on-disk/weight size AND precision from the HF card
(
Precision: BF16= 2 bytes/param; FP8/Q8 ≈ 1 byte; Q4 ≈ 0.5 byte). 35B params: BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB. - Compare against the HOST'S USABLE RAM (not just VRAM) for runtimes that stream weights from RAM (FreeToken host-RAM experts, llama.cpp CPU offload, PP/worker splits).
- Budget RAM for weights + KV pool + OS + page cache. If weights alone exceed RAM, ABORT — pool tuning shrinks KV, it does not shrink weights.
The self-sustaining thrash (why 'it'll just OOM and recover' is wrong)
- Large swap (tens of GB) → kernel PAGES instead of OOM-killing → sshd/user space starved indefinitely; no recovery window opens by itself.
- systemd
Restart=on-failure|always→ crash-reload-rethrash loop. - Unit
enabled+WantedBy=multi-user.target→ a reboot re-thrashes on boot until the bad config is masked/reverted. A plain power-cycle does NOT fix it.
Recovery (needs a shell; may need console / SysRq-to-GRUB)
sudo systemctl mask --now <svc> # stop + break the reboot trap FIRST
cd <checkout> && git fetch origin && git reset --hard origin/main # pick up revert
grep MODEL_DIR <env> # confirm fixed target
sudo systemctl unmask <svc> && sudo systemctl restart <svc>
curl -s <health> # then verify with a REAL completion
Deploy-script caveat
deploy.sh/health-gate 'healthy after Ns' = systemd ACTIVE, not model LOADED.
Always confirm load with a real completion (curl /v1/chat/completions), not
status alone.
GPU quant note
RTX 3090 (Ampere) has no FP8/FP4 tensor cores: FP8 weights run via upcast but still must fit in RAM/VRAM. Choose quant by SIZE fit, then accept the compute path the card supports.