---
name: llm-model-swap-fit-preflight
description: "Preflight an LLM model swap for hardware fit before deploying; catch the silent swap-thrash outage mode (large RAM OOM, self-restarting + boot-enabled units) and recover."
---

# LLM Model-Swap Fit Preflight

Never deploy a model swap without confirming the checkpoint FITS, and know the
failure mode when it doesn't: it is a silent, self-sustaining thrash, not a crash.

## Preflight (before deploy)

1. Get the checkpoint's on-disk/weight size AND precision from the HF card
   (`Precision: BF16` = 2 bytes/param; FP8/Q8 ≈ 1 byte; Q4 ≈ 0.5 byte).
   35B params: BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
2. Compare against the HOST'S USABLE RAM (not just VRAM) for runtimes that
   stream weights from RAM (FreeToken host-RAM experts, llama.cpp CPU offload,
   PP/worker splits).
3. Budget RAM for weights + KV pool + OS + page cache. If weights alone exceed
   RAM, ABORT — pool tuning shrinks KV, it does not shrink weights.

## The self-sustaining thrash (why 'it'll just OOM and recover' is wrong)

- Large swap (tens of GB) → kernel PAGES instead of OOM-killing → sshd/user
  space starved indefinitely; no recovery window opens by itself.
- systemd `Restart=on-failure|always` → crash-reload-rethrash loop.
- Unit `enabled` + `WantedBy=multi-user.target` → a reboot re-thrashes on boot
  until the bad config is masked/reverted. A plain power-cycle does NOT fix it.

## Recovery (needs a shell; may need console / SysRq-to-GRUB)

```
sudo systemctl mask --now <svc>        # stop + break the reboot trap FIRST
cd <checkout> && git fetch origin && git reset --hard origin/main   # pick up revert
grep MODEL_DIR <env>                   # confirm fixed target
sudo systemctl unmask <svc> && sudo systemctl restart <svc>
curl -s <health>                        # then verify with a REAL completion
```

## Deploy-script caveat

`deploy.sh`/health-gate 'healthy after Ns' = systemd ACTIVE, not model LOADED.
Always confirm load with a real completion (curl /v1/chat/completions), not
status alone.

## GPU quant note

RTX 3090 (Ampere) has no FP8/FP4 tensor cores: FP8 weights run via upcast but
still must fit in RAM/VRAM. Choose quant by SIZE fit, then accept the compute
path the card supports.