Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
51 lines
2.2 KiB
Markdown
51 lines
2.2 KiB
Markdown
---
|
||
name: llm-model-swap-fit-preflight
|
||
description: "Preflight an LLM model swap for hardware fit before deploying; catch the silent swap-thrash outage mode (large RAM OOM, self-restarting + boot-enabled units) and recover."
|
||
---
|
||
|
||
# LLM Model-Swap Fit Preflight
|
||
|
||
Never deploy a model swap without confirming the checkpoint FITS, and know the
|
||
failure mode when it doesn't: it is a silent, self-sustaining thrash, not a crash.
|
||
|
||
## Preflight (before deploy)
|
||
|
||
1. Get the checkpoint's on-disk/weight size AND precision from the HF card
|
||
(`Precision: BF16` = 2 bytes/param; FP8/Q8 ≈ 1 byte; Q4 ≈ 0.5 byte).
|
||
35B params: BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
|
||
2. Compare against the HOST'S USABLE RAM (not just VRAM) for runtimes that
|
||
stream weights from RAM (FreeToken host-RAM experts, llama.cpp CPU offload,
|
||
PP/worker splits).
|
||
3. Budget RAM for weights + KV pool + OS + page cache. If weights alone exceed
|
||
RAM, ABORT — pool tuning shrinks KV, it does not shrink weights.
|
||
|
||
## The self-sustaining thrash (why 'it'll just OOM and recover' is wrong)
|
||
|
||
- Large swap (tens of GB) → kernel PAGES instead of OOM-killing → sshd/user
|
||
space starved indefinitely; no recovery window opens by itself.
|
||
- systemd `Restart=on-failure|always` → crash-reload-rethrash loop.
|
||
- Unit `enabled` + `WantedBy=multi-user.target` → a reboot re-thrashes on boot
|
||
until the bad config is masked/reverted. A plain power-cycle does NOT fix it.
|
||
|
||
## Recovery (needs a shell; may need console / SysRq-to-GRUB)
|
||
|
||
```
|
||
sudo systemctl mask --now <svc> # stop + break the reboot trap FIRST
|
||
cd <checkout> && git fetch origin && git reset --hard origin/main # pick up revert
|
||
grep MODEL_DIR <env> # confirm fixed target
|
||
sudo systemctl unmask <svc> && sudo systemctl restart <svc>
|
||
curl -s <health> # then verify with a REAL completion
|
||
```
|
||
|
||
## Deploy-script caveat
|
||
|
||
`deploy.sh`/health-gate 'healthy after Ns' = systemd ACTIVE, not model LOADED.
|
||
Always confirm load with a real completion (curl /v1/chat/completions), not
|
||
status alone.
|
||
|
||
## GPU quant note
|
||
|
||
RTX 3090 (Ampere) has no FP8/FP4 tensor cores: FP8 weights run via upcast but
|
||
still must fit in RAM/VRAM. Choose quant by SIZE fit, then accept the compute
|
||
path the card supports.
|