--- type: reference title: Model Swap and Rollback description: "The model-swap discipline: fit preflight, fetch verification, build and flag checks, live validation, three-pin revert." tags: [models, swap, gguf, huggingface, rollback, deployment, verification] timestamp: 2026-10-05T00:00:00-07:00 --- # Model Swap and Rollback Applies to every endpoint swap on the fleet (frank-zealot / frank-dragoon / frank-archon). Repo law: all config flows through git; the live models are host state under `/home/dank/models//`. ## 1. Fit preflight — never deploy a swap without confirming it FITS 1. Get the checkpoint's on-disk/weight size AND precision from the HF card (`Precision: BF16` = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params: BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB. 2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload, layer splits). Budget weights + KV pool + OS + page cache; **if weights alone exceed RAM, ABORT** — pool tuning shrinks KV, it does not shrink weights. 3. **The failure mode is a self-sustaining thrash, not a crash:** a large swap makes the kernel PAGE instead of OOM-killing → sshd/user space starved indefinitely, no recovery window opens by itself; systemd `Restart=always` turns it into a crash-reload-rethrash loop; `enabled` + `WantedBy=multi-user.target` re-thrashes on every reboot. A power cycle does NOT fix it. Recovery (may need console / SysRq-to-GRUB): ```bash sudo systemctl mask --now # break the reboot trap FIRST cd && git fetch origin && git reset --hard origin/main grep MODEL_DIR # confirm the fixed target sudo systemctl unmask && sudo systemctl restart curl -s # then a REAL completion ``` Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED — confirm with a real `/v1/chat/completions`. On Ampere an FP8 file buys memory and no compute ([[Quantization by Hardware Architecture]]): choose the quant by size fit, then accept the compute path the card supports. Before choosing a smaller quant, state WHICH of the two you are buying — headroom, or a design change (a smaller quant does not change what the runtime can express; per-tensor cross-device placement stays refused under `--override-tensor`, and a layer split moves only contiguous layer ranges). ## 2. Ground truth first - Read the HF model card (runtime pin, hard constraints, recommended args) + the repo trio `scripts/.env`, `scripts/serve-.sh`, `scripts/.service` + the runbook under `serverAppConfigs-docs/LLM/`. - Check the host before downloading anything: `ls /home/dank/models/`, `df -h /home`, `du -sh` — bench-era or prior-deployment files often already exist. ## 3. Fetch and verify (HF payloads) **Inventory by QUANT NAME, never by extension** — large quants ship as `.partNofM` fragments, so an `endswith(".gguf")` filter makes a whole rung look nonexistent (AGENTS.md carries the worked fragment ledger). Check both mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors return 403 even with a valid token until the account accepts the gate — prefer the ungated copy. ```bash # token out of band: mode-600 curl config; a token passed as -H is world-readable printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc chmod 600 /tmp/hf.curlrc curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name" ``` `-f` is MANDATORY: `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes the error body as the artifact (identical 153-byte shards, cheerful DONE, zero bytes of model). Never `pgrep -a` a running fetch (token is in `/proc//cmdline`); watch progress by size: `while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done`. **Verify, then assemble:** per part, byte count == Hub's declared size AND `sha256sum` == `lfs.oid`; concatenate in NUMERIC order; assembled total == sum of parts (compute the sum with python, not by hand) and `head -c 4` == `GGUF`. The assembled file has no published hash — the per-part oids + exact total are the chain. Keep the fragments when disk allows (only copies re-verifiable against the Hub); failure artifacts and the replaced payload go to `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm` — upstreams DELIST quants and the local file may be the last copy. Fetch-script one-liners: dedicated scripts (`scripts/download-*.sh`) already do `-f` + byte-count + LFS sha256 + `--verify-only`; ad-hoc fetches tend to skip all three. HF cache snapshots SYMLINK every file into `blobs/` and those links dangle inside containers — serve scripts and the deployer refuse symlinks and missing tokenizer files for exactly this reason. ## 4. Runtime build (when the card pins a fork/build) - Clone + build with the card's exact cmake line, plus `-DLLAMA_BUILD_WEBUI=OFF` — WebUI asset provisioning (npm / HF llama-ui bucket) aborts llama.cpp link at 98 % on these hosts; the API server is unaffected. - Verify every flag the serve script passes exists in the new binary BEFORE touching the service: `llama-server --help | grep -q -- ""` for each; list MISSes. Read the GGUF's tensor histogram first if the payload uses a custom type — the FORMAT picks the runtime ([[GGUF Compatibility]]). ## 5. Config via git only Edit `scripts/.env` (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve script header/defaults locally; `bash -n`; `./tests/run.sh` must pass; commit; push; `git pull --ff-only` on the target. Never edit configs on the remote. Renaming a model dir (`mv`) is host state and allowed. The env-file hook pattern (`FRANK_*_ENV_FILE`) is what makes bench windows and staged arms possible — keep it intact. ## 6. Live validation - No scratch ports — one GPU cannot host two big models. Ensure the old unit is truly gone first: `systemctl is-active` can hang in `stop-sigterm` (llama.cpp never preempts an in-flight generation); poll until the PID exits. - Manual serve for smoke: `setsid nohup bash scripts/serve-.sh > /tmp/x.log 2>&1 &` — plain `ssh 'nohup … &'` dies with its ssh session. - Verify: health 200; a REAL `/v1/chat/completions`; reasoning flows (`reasoning_content` non-empty); if the endpoint takes images, a solid-color probe through an `image_url` content part (legacy top-level `images` is silently ignored; probe with ≥256 px). - vLLM specifics: first request after boot can take ~18 s (deferred Triton loads) — warm and discard; `--reasoning-parser` / tool parsers are part of the contract ([[vLLM Boot Forensics and Config Interrogation]]). ## 7. systemd handoff — THE TRAP `./scripts/deploy.sh --service ` gates health, but a leftover manual server holding the port makes the health check pass while the UNIT crash-loops on bind. Always confirm ownership: `systemctl show -p MainPID` equals the single `pgrep -x llama-server` (or the container runtime PID); only then trust "healthy". If a real long-context task is queued, do the handoff anyway — client aborts are recoverable, drift is not. ## 8. Close-out (mandatory) 1. Docs: active runbook updated (build recipe, profile, validation numbers incl. real-workload prefill/decode); superseded runbook gets a RETIRED banner and stays as the rollback record; `LLM/index.md` mermaid + active list; AGENTS.md fleet table + runtime section + history chain; work-log entry in `serverAppConfigs-docs/log.md`. 2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides the pool; e.g. a 524288 pool over 2 slots = 262144 per slot); vLLM reports `max-model-len` and its concurrency line. Overselling causes silent client-side overflow or a discovered-wall 400. Also `maxTokens` vs the engine's cap, `input` capabilities (text/image), and `reasoningContentField` per runtime (llama.cpp/SGLang emit `reasoning_content`, vLLM emits `reasoning`). Fix every `modelRoles` entry that named the old endpoint. 3. **Rollback = one env commit** (paths + LLAMA_BIN + flags), prior binary + payload kept on disk, rollback numbers recorded in the runbook. ## 9. Revert-pin an ad hoc experiment (the three pins) A campaign is "done" only when all three exist — without them a `podman pull`, an image update, or a lost scratch script destroys the way back: 1. **Git tag (annotated, pushed):** `git tag -a -tuned-YYYYMMDD -m ""` — a future agent must reconstruct the campaign from the tag message alone. 2. **Image digest in the serve script:** floating tags move. Capture `podman inspect --format '{{.ImageName}} {{.Image}}'` + `podman images --digests`, patch `IMAGE=` to `repo@sha256:` with a comment (image ID, date, old ref), verify `podman image exists repo@sha256:` and `bash -n`. Digest refs survive upstream pushes; `podman pull repo@sha256:` re-fetches exact bytes. One host can hold MULTIPLE images of one repo — pin by the digest of the RUNNING container, never by tag name. Never edit the pinned line back to a floating tag; new experiments branch from the digest. 3. **Local alias:** `podman tag -runtime:tuned-YYYYMMDD`. Record scratch-script paths + full flag surface in the runbook (hook env vars in CAPS); expired images/scripts go to `.AGENTTRASH`, never `rm`. Pure measurement sessions that changed no state need only pin 1. ## 10. Research verification discipline (for "can model X run here?") - Scouted research summaries have shipped fabricated citations on this fleet (invented HF repos, plausible-but-wrong PR numbers, stale engine versions). NEVER copy a scout's citations into a deliverable — re-verify each load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it enters anything. - The in-repo runbooks are FIRST-HAND evidence and outrank any web claim about our own hosts. - Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active, CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path (heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K). GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind `-c`, the default hard-hung a 128 GB host; native FP8 328.3 GB; `-fa off` MANDATORY, flash-attn breaks MLA correctness; llama.cpp arch `glm5next` lived in unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp MMLU-Pro vs Q8_0). - GGUF parsing gotcha: expert tensors are `blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`; a `len(parts) == 3` filter matches zero layers with no parse error. Filter on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`. ## 11. Adding a second LLM endpoint to a host When a host must serve two models at once (an iGPU runtime plus a second card), or one endpoint is moving and a new one takes the freed hardware. **0. Decide the shape before touching hardware.** One provider per HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs) — a second endpoint is a new provider block, not a second model in the existing one. Read the budget as an inequality: for unified memory the engine's own fit check is the arbiter (`budget = host − pinned weights − reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV by counting only full-attention layers — and remember `--parallel N` DIVIDES `--ctx-size`. A co-tenant is a budget line, not a footnote: free RAM by changing the POOL or the ARENA (the reserve is a fit-check term, never an allocation) and drop optional charges (a vision tower is ~2 GiB) rather than hoping. **1. Repo first, and retire in the same commit.** New service files (`scripts/.env`, `.service`, `serve-.sh` with hard guards for missing artifacts); planner entries (`SERVICE_ORDER` + the target's service set + path mappings, inventories in `deploy.sh`/`deploy-remote.sh`, the dispatch case); retire the obsoleted pieces into `.AGENTTRASH/` with a suffix — and check what pointed at them: bench harnesses, a chat template only the old runtime read, and `scripts/test-changed.sh` mappings are the usual victims (a mapping to a retired test file fails the pre-commit wrapper silently for weeks). One bats file per service asserting INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation), not constants; a negative grep must strip comments first (`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining the very thing you removed. **2. The port-bind race (rootless podman).** pasta binds the published port at container CREATE; a just-stopped predecessor still holds it. Order the handler: retire the old manager's unit → stop THIS service's own unit (no-op on first cutover) → wait for every socket on the port to clear ([[Container Host Reachability]]) → install → start. Without the second step a redeploy deadlocks against its own gate. **3. Wire the surfaces, refuse impossible signals.** Point the harness small/fast + image roles at the new endpoint; move the vision role WITH the vision tower (one owner per capability). `shared/homepage/services.json`: a card per endpoint, probed from the monitor's own host. Prometheus: only add a job if the runtime actually exposes Prometheus text — and re-measure rather than repeat, because that answer moves: halogen's HTTP surface has grown twice since this page was written, and the endpoint list that matters for a scrape target — including whether `/metrics` answers — is [[frank-dragoon]]'s wire contract. **4. Cutover, then verify by driving it.** `deploy.sh --service --service ` (order matters); per endpoint: `/health`, `/v1/models` reports the alias, a chat completion returns content AND `reasoning_content`, a `tools` request returns `tool_calls`, an image request is answered (or refused with a named reason), the runtime's own speculation counters move, and the unit is `enabled` and `active`. All three LLM endpoints are **root** units today (`scripts/frank-archon.service`, `scripts/frank-dragoon.service`, `scripts/frank-zealot.service`), so linger is not what gates them; linger matters for rootless quadlets. Read the engine's boot ledger for a silent clamp. Finally spend one real request through the harness per changed role — routing labels lie; the servers' own journals do not. Traps that look like model faults: `500 tools param requires --jinja flag` (runtime lacks the Jinja path — pin it); `500 Jinja Exception: Unexpected reasoning effort ` (fold via `compat.reasoningEffortMap`); the same request works by hand but a subagent dies (subagents run at a different thinking level — check both); the container dies instantly although the predecessor "stopped" (the port race above). ## Related - [[GGUF Compatibility]] — which runtime can load a payload at all - [[Llama Cpp Serving Sizing]] — pricing the shape once it loads - [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic - [[LLM Bench Windows and Measurement]] — measuring the new deployment * [[frank-archon]] — the live Dankserver endpoint; the tcclaviger vLLM era (2026-09-29 → 10-05) is a row in its Eras table