chore(skills): promote runbook skills to managed-skills; drop superseded SKILL.md
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
This commit is contained in:
1 parent
7be39d60dd
commit
9bfa0be85d
48 files changed
+1361
-42
No files matched your search
@@ -0,0 +1,50 @@
|
||||
---
|
||||
name: llm-model-swap-fit-preflight
|
||||
description: "Preflight an LLM model swap for hardware fit before deploying; catch the silent swap-thrash outage mode (large RAM OOM, self-restarting + boot-enabled units) and recover."
|
||||
---
|
||||
|
||||
# LLM Model-Swap Fit Preflight
|
||||
|
||||
Never deploy a model swap without confirming the checkpoint FITS, and know the
|
||||
failure mode when it doesn't: it is a silent, self-sustaining thrash, not a crash.
|
||||
|
||||
## Preflight (before deploy)
|
||||
|
||||
1. Get the checkpoint's on-disk/weight size AND precision from the HF card
|
||||
(`Precision: BF16` = 2 bytes/param; FP8/Q8 ≈ 1 byte; Q4 ≈ 0.5 byte).
|
||||
35B params: BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
|
||||
2. Compare against the HOST'S USABLE RAM (not just VRAM) for runtimes that
|
||||
stream weights from RAM (FreeToken host-RAM experts, llama.cpp CPU offload,
|
||||
PP/worker splits).
|
||||
3. Budget RAM for weights + KV pool + OS + page cache. If weights alone exceed
|
||||
RAM, ABORT — pool tuning shrinks KV, it does not shrink weights.
|
||||
|
||||
## The self-sustaining thrash (why 'it'll just OOM and recover' is wrong)
|
||||
|
||||
- Large swap (tens of GB) → kernel PAGES instead of OOM-killing → sshd/user
|
||||
space starved indefinitely; no recovery window opens by itself.
|
||||
- systemd `Restart=on-failure|always` → crash-reload-rethrash loop.
|
||||
- Unit `enabled` + `WantedBy=multi-user.target` → a reboot re-thrashes on boot
|
||||
until the bad config is masked/reverted. A plain power-cycle does NOT fix it.
|
||||
|
||||
## Recovery (needs a shell; may need console / SysRq-to-GRUB)
|
||||
|
||||
```
|
||||
sudo systemctl mask --now <svc> # stop + break the reboot trap FIRST
|
||||
cd <checkout> && git fetch origin && git reset --hard origin/main # pick up revert
|
||||
grep MODEL_DIR <env> # confirm fixed target
|
||||
sudo systemctl unmask <svc> && sudo systemctl restart <svc>
|
||||
curl -s <health> # then verify with a REAL completion
|
||||
```
|
||||
|
||||
## Deploy-script caveat
|
||||
|
||||
`deploy.sh`/health-gate 'healthy after Ns' = systemd ACTIVE, not model LOADED.
|
||||
Always confirm load with a real completion (curl /v1/chat/completions), not
|
||||
status alone.
|
||||
|
||||
## GPU quant note
|
||||
|
||||
RTX 3090 (Ampere) has no FP8/FP4 tensor cores: FP8 weights run via upcast but
|
||||
still must fit in RAM/VRAM. Choose quant by SIZE fit, then accept the compute
|
||||
path the card supports.
|
||||
Reference in new issue
Block a user