serverAppConfigs/serverAppConfigs-docs/LLM/Model Swap and Rollback.md

---
type: reference
title: Model Swap and Rollback
description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
timestamp: 2026-10-05T00:00:00-07:00
---

# Model Swap and Rollback

Applies to every endpoint swap on the fleet (frank-small / frank-large /
frank-primary). Repo law: all config flows through git; the live models are
host state under `/home/dank/models/<name>/`.

## 1. Fit preflight — never deploy a swap without confirming it FITS

1. Get the checkpoint's on-disk/weight size AND precision from the HF card
   (`Precision: BF16` = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params:
   BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that
   stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload,
   layer splits). Budget weights + KV pool + OS + page cache; **if weights
   alone exceed RAM, ABORT** — pool tuning shrinks KV, it does not shrink
   weights.
3. **The failure mode is a self-sustaining thrash, not a crash:** a large
   swap makes the kernel PAGE instead of OOM-killing → sshd/user space
   starved indefinitely, no recovery window opens by itself; systemd
   `Restart=always` turns it into a crash-reload-rethrash loop; `enabled` +
   `WantedBy=multi-user.target` re-thrashes on every reboot. A power cycle
   does NOT fix it.

Recovery (may need console / SysRq-to-GRUB):

```bash
sudo systemctl mask --now <svc>                       # break the reboot trap FIRST
cd <checkout> && git fetch origin && git reset --hard origin/main
grep MODEL_DIR <env>                                  # confirm the fixed target
sudo systemctl unmask <svc> && sudo systemctl restart <svc>
curl -s <health>                                      # then a REAL completion
```

Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
quant by size fit, then accept the compute path the card supports. Before
choosing a smaller quant, state WHICH of the two you are buying — headroom,
or a design change (a smaller quant does not change what the runtime can
express; per-tensor cross-device placement stays refused under
`--override-tensor`, and a layer split moves only contiguous layer ranges).

## 2. Ground truth first

- Read the HF model card (runtime pin, hard constraints, recommended args)
  + the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
  `scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
- Check the host before downloading anything: `ls /home/dank/models/`,
  `df -h /home`, `du -sh` — bench-era or prior-deployment files often already
  exist.

## 3. Fetch and verify (HF payloads)

**Inventory by QUANT NAME, never by extension** — large quants ship as
`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
return 403 even with a valid token until the account accepts the gate —
prefer the ungated copy.

```bash
# token out of band: mode-600 curl config; a token passed as -H is world-readable
printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
chmod 600 /tmp/hf.curlrc
curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name"
```

`-f` is MANDATORY: `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes
the error body as the artifact (identical 153-byte shards, cheerful DONE,
zero bytes of model). Never `pgrep -a` a running fetch (token is in
`/proc/<pid>/cmdline`); watch progress by size:
`while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done`.

**Verify, then assemble:** per part, byte count == Hub's declared size AND
`sha256sum` == `lfs.oid`; concatenate in NUMERIC order; assembled total ==
sum of parts (compute the sum with python, not by hand) and `head -c 4` ==
`GGUF`. The assembled file has no published hash — the per-part oids + exact
total are the chain. Keep the fragments when disk allows (only copies
re-verifiable against the Hub); failure artifacts and the replaced payload
go to `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm` — upstreams DELIST quants
and the local file may be the last copy.

Fetch-script one-liners: dedicated scripts (`scripts/download-*.sh`) already
do `-f` + byte-count + LFS sha256 + `--verify-only`; ad-hoc fetches tend to
skip all three. HF cache snapshots SYMLINK every file into `blobs/` and
those links dangle inside containers — serve scripts and the deployer refuse
symlinks and missing tokenizer files for exactly this reason.

## 4. Runtime build (when the card pins a fork/build)

- Clone + build with the card's exact cmake line, plus
  `-DLLAMA_BUILD_WEBUI=OFF` — WebUI asset provisioning (npm / HF llama-ui
  bucket) aborts llama.cpp link at 98 % on these hosts; the API server is
  unaffected.
- Verify every flag the serve script passes exists in the new binary BEFORE
  touching the service: `llama-server --help | grep -q -- "<flag>"` for each;
  list MISSes. Read the GGUF's tensor histogram first if the payload uses a
  custom type — the FORMAT picks the runtime ([[GGUF Compatibility]]).

## 5. Config via git only

Edit `scripts/<svc>.env` (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve
script header/defaults locally; `bash -n`; `./tests/run.sh` must pass;
commit; push; `git pull --ff-only` on the target. Never edit configs on the
remote. Renaming a model dir (`mv`) is host state and allowed. The env-file
hook pattern (`FRANK_*_ENV_FILE`) is what makes bench windows and staged
arms possible — keep it intact.

## 6. Live validation

- No scratch ports — one GPU cannot host two big models. Ensure the old unit
  is truly gone first: `systemctl is-active` can hang in `stop-sigterm`
  (llama.cpp never preempts an in-flight generation); poll until the PID
  exits.
- Manual serve for smoke: `setsid nohup bash scripts/serve-<svc>.sh >
  /tmp/x.log 2>&1 &` — plain `ssh 'nohup … &'` dies with its ssh session.
- Verify: health 200; a REAL `/v1/chat/completions`; reasoning flows
  (`reasoning_content` non-empty); if the endpoint takes images, a
  solid-color probe through an `image_url` content part (legacy top-level
  `images` is silently ignored; probe with ≥256 px).
- vLLM specifics: first request after boot can take ~18 s (deferred Triton
  loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
  the contract ([[vLLM Boot Forensics and Config Interrogation]]).

## 7. systemd handoff — THE TRAP

`./scripts/deploy.sh <host> --service <svc>` gates health, but a leftover
manual server holding the port makes the health check pass while the UNIT
crash-loops on bind. Always confirm ownership:
`systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server`
(or the container runtime PID); only then trust "healthy". If a real
long-context task is queued, do the handoff anyway — client aborts are
recoverable, drift is not.

## 8. Close-out (mandatory)

1. Docs: active runbook updated (build recipe, profile, validation numbers
   incl. real-workload prefill/decode); superseded runbook gets a RETIRED
   banner and stays as the rollback record; `LLM/index.md` mermaid + active
   list; AGENTS.md fleet table + runtime section + history chain; work-log
   entry in `serverAppConfigs-docs/log.md`.
2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
   context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
   the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
   `max-model-len` and its concurrency line. Overselling causes silent
   client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
   engine's cap, `input` capabilities (text/image), and
   `reasoningContentField` per runtime (llama.cpp/SGLang emit
   `reasoning_content`, vLLM emits `reasoning`). Fix every `modelRoles` entry
   that named the old endpoint.
3. **Rollback = one env commit** (paths + LLAMA_BIN + flags), prior binary +
   payload kept on disk, rollback numbers recorded in the runbook.

## 9. Revert-pin an ad hoc experiment (the three pins)

A campaign is "done" only when all three exist — without them a
`podman pull`, an image update, or a lost scratch script destroys the way
back:

1. **Git tag (annotated, pushed):**
   `git tag -a <exp>-tuned-YYYYMMDD -m "<tuned env/flags + measured headline +
   runbook path + revert command>"` — a future agent must reconstruct the
   campaign from the tag message alone.
2. **Image digest in the serve script:** floating tags move. Capture
   `podman inspect <ctr> --format '{{.ImageName}} {{.Image}}'` + `podman
   images --digests`, patch `IMAGE=` to `repo@sha256:<repodigest>` with a
   comment (image ID, date, old ref), verify `podman image exists
   repo@sha256:<digest>` and `bash -n`. Digest refs survive upstream pushes;
   `podman pull repo@sha256:<digest>` re-fetches exact bytes. One host can
   hold MULTIPLE images of one repo — pin by the digest of the RUNNING
   container, never by tag name. Never edit the pinned line back to a
   floating tag; new experiments branch from the digest.
3. **Local alias:** `podman tag <id> <exp>-runtime:tuned-YYYYMMDD`.

Record scratch-script paths + full flag surface in the runbook (hook env
vars in CAPS); expired images/scripts go to `.AGENTTRASH`, never `rm`.
Pure measurement sessions that changed no state need only pin 1.

## 10. Research verification discipline (for "can model X run here?")

- Scouted research summaries have shipped fabricated citations on this fleet
  (invented HF repos, plausible-but-wrong PR numbers, stale engine versions).
  NEVER copy a scout's citations into a deliverable — re-verify each
  load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it
  enters anything.
- The in-repo runbooks are FIRST-HAND evidence and outrank any web claim
  about our own hosts.
- Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active,
  CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early
  GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path
  (heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K).
  GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind `-c`, the
  default hard-hung a 128 GB host; native FP8 328.3 GB; `-fa off` MANDATORY,
  flash-attn breaks MLA correctness; llama.cpp arch `glm5next` lived in
  unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant
  ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
  MMLU-Pro vs Q8_0).
- GGUF parsing gotcha: expert tensors are
  `blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
  a `len(parts) == 3` filter matches zero layers with no parse error. Filter
  on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.

## 11. Adding a second LLM endpoint to a host

When a host must serve two models at once (an iGPU runtime plus a second
card), or one endpoint is moving and a new one takes the freed hardware.

**0. Decide the shape before touching hardware.** One provider per
HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
— a second endpoint is a new provider block, not a second model in the
existing one. Read the budget as an inequality: for unified memory the
engine's own fit check is the arbiter (`budget = host − pinned weights −
reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV
by counting only full-attention layers — and remember `--parallel N`
DIVIDES `--ctx-size`. A co-tenant is a budget line, not a footnote: free
RAM by changing the POOL or the ARENA (the reserve is a fit-check term,
never an allocation) and drop optional charges (a vision tower is ~2 GiB)
rather than hoping.

**1. Repo first, and retire in the same commit.** New service files
(`scripts/<svc>.env`, `.service`, `serve-<svc>.sh` with hard guards for
missing artifacts); planner entries (`SERVICE_ORDER` + the target's service
set + path mappings, inventories in `deploy.sh`/`deploy-remote.sh`, the
dispatch case); retire the obsoleted pieces into `.AGENTTRASH/` with a
suffix — and check what pointed at them: bench harnesses, a chat template
only the old runtime read, and `scripts/test-changed.sh` mappings are the
usual victims (a mapping to a retired test file fails the pre-commit
wrapper silently for weeks). One bats file per service asserting
INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
not constants; a negative grep must strip comments first
(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
the very thing you removed.

**2. The port-bind race (rootless podman).** pasta binds the published port
at container CREATE; a just-stopped predecessor still holds it. Order the
handler: retire the old manager's unit → stop THIS service's own unit
(no-op on first cutover) → wait for every socket on the port to clear
([[Container Host Reachability]]) → install → start. Without the second
step a redeploy deadlocks against its own gate.

**3. Wire the surfaces, refuse impossible signals.** Point the harness
small/fast + image roles at the new endpoint; move the vision role WITH the
vision tower (one owner per capability). `shared/homepage/services.json`: a
card per endpoint, probed from the monitor's own host. Prometheus: only add
a job if the runtime actually exposes Prometheus text (halogen publishes
`/health`, `/cache`, `/v1/*` and nothing else — the job was REMOVED with the
reason written down rather than left as a permanent red).

**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
<large> --service <small>` (order matters); per endpoint: `/health`,
`/v1/models` reports the alias, a chat completion returns content AND
`reasoning_content`, a `tools` request returns `tool_calls`, an image
request is answered (or refused with a named reason), the runtime's own
speculation counters move, and the unit is `enabled`/`generated` with linger
on. Read the engine's boot ledger for a silent clamp. Finally spend one
real request through the harness per changed role — routing labels lie; the
servers' own journals do not.

Traps that look like model faults: `500 tools param requires --jinja flag`
(runtime lacks the Jinja path — pin it); `500 Jinja Exception: Unexpected
reasoning effort <level>` (fold via `compat.reasoningEffortMap`); the same
request works by hand but a subagent dies (subagents run at a different
thinking level — check both); the container dies instantly although the
predecessor "stopped" (the port race above).

## Related

- [[GGUF Compatibility]] — which runtime can load a payload at all
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint