serverAppConfigs/serverAppConfigs-docs/LLM/Model Swap and Rollback.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Model Swap and Rollback	The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.	models swap gguf huggingface rollback deployment verification	2026-10-05T00:00:00-07:00

Model Swap and Rollback

Applies to every endpoint swap on the fleet (frank-small / frank-large / frank-primary). Repo law: all config flows through git; the live models are host state under /home/dank/models/<name>/.

1. Fit preflight — never deploy a swap without confirming it FITS

1. Get the checkpoint's on-disk/weight size AND precision from the HF card (Precision: BF16 = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params: BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload, layer splits). Budget weights + KV pool + OS + page cache; if weights alone exceed RAM, ABORT — pool tuning shrinks KV, it does not shrink weights.
3. The failure mode is a self-sustaining thrash, not a crash: a large swap makes the kernel PAGE instead of OOM-killing → sshd/user space starved indefinitely, no recovery window opens by itself; systemd Restart=always turns it into a crash-reload-rethrash loop; enabled + WantedBy=multi-user.target re-thrashes on every reboot. A power cycle does NOT fix it.

Recovery (may need console / SysRq-to-GRUB):

sudo systemctl mask --now <svc>                       # break the reboot trap FIRST
cd <checkout> && git fetch origin && git reset --hard origin/main
grep MODEL_DIR <env>                                  # confirm the fixed target
sudo systemctl unmask <svc> && sudo systemctl restart <svc>
curl -s <health>                                      # then a REAL completion

Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED — confirm with a real /v1/chat/completions. RTX 3090 (Ampere) has no FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose quant by size fit, then accept the compute path the card supports. Before choosing a smaller quant, state WHICH of the two you are buying — headroom, or a design change (a smaller quant does not change what the runtime can express; per-tensor cross-device placement stays refused under --override-tensor, and a layer split moves only contiguous layer ranges).

2. Ground truth first

• Read the HF model card (runtime pin, hard constraints, recommended args)
  • the repo trio scripts/<svc>.env, scripts/serve-<svc>.sh, scripts/<svc>.service + the runbook under serverAppConfigs-docs/LLM/.
• Check the host before downloading anything: ls /home/dank/models/, df -h /home, du -sh — bench-era or prior-deployment files often already exist.

3. Fetch and verify (HF payloads)

Inventory by QUANT NAME, never by extension — large quants ship as .partNofM fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an endswith(".gguf") filter makes Q4_K_S look nonexistent. Check both mradermacher repos (…-i1-GGUF imatrix, …-GGUF plain) and the model's own repo; the tree API's blobs=true gives lfs.oid per file. Gated mirrors return 403 even with a valid token until the account accepts the gate — prefer the ungated copy.

# token out of band: mode-600 curl config; a token passed as -H is world-readable
printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
chmod 600 /tmp/hf.curlrc
curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name"

-f is MANDATORY: curl -o FILE URL exits 0 on HTTP 401/403 and writes the error body as the artifact (identical 153-byte shards, cheerful DONE, zero bytes of model). Never pgrep -a a running fetch (token is in /proc/<pid>/cmdline); watch progress by size: while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done.

Verify, then assemble: per part, byte count == Hub's declared size AND sha256sum == lfs.oid; concatenate in NUMERIC order; assembled total == sum of parts (compute the sum with python, not by hand) and head -c 4 == GGUF. The assembled file has no published hash — the per-part oids + exact total are the chain. Keep the fragments when disk allows (only copies re-verifiable against the Hub); failure artifacts and the replaced payload go to ~/.AGENTTRASH/YYYYMMDD-topic, never rm — upstreams DELIST quants and the local file may be the last copy.

Fetch-script one-liners: dedicated scripts (scripts/download-*.sh) already do -f + byte-count + LFS sha256 + --verify-only; ad-hoc fetches tend to skip all three. HF cache snapshots SYMLINK every file into blobs/ and those links dangle inside containers — serve scripts and the deployer refuse symlinks and missing tokenizer files for exactly this reason.

4. Runtime build (when the card pins a fork/build)

• Clone + build with the card's exact cmake line, plus -DLLAMA_BUILD_WEBUI=OFF — WebUI asset provisioning (npm / HF llama-ui bucket) aborts llama.cpp link at 98 % on these hosts; the API server is unaffected.
• Verify every flag the serve script passes exists in the new binary BEFORE touching the service: llama-server --help | grep -q -- "<flag>" for each; list MISSes. Read the GGUF's tensor histogram first if the payload uses a custom type — the FORMAT picks the runtime (GGUF Compatibility).

5. Config via git only

Edit scripts/<svc>.env (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve script header/defaults locally; bash -n; ./tests/run.sh must pass; commit; push; git pull --ff-only on the target. Never edit configs on the remote. Renaming a model dir (mv) is host state and allowed. The env-file hook pattern (FRANK_*_ENV_FILE) is what makes bench windows and staged arms possible — keep it intact.

6. Live validation

• No scratch ports — one GPU cannot host two big models. Ensure the old unit is truly gone first: systemctl is-active can hang in stop-sigterm (llama.cpp never preempts an in-flight generation); poll until the PID exits.
• Manual serve for smoke: setsid nohup bash scripts/serve-<svc>.sh > /tmp/x.log 2>&1 & — plain ssh 'nohup … &' dies with its ssh session.
• Verify: health 200; a REAL /v1/chat/completions; reasoning flows (reasoning_content non-empty); if the endpoint takes images, a solid-color probe through an image_url content part (legacy top-level images is silently ignored; probe with ≥256 px).
• vLLM specifics: first request after boot can take ~18 s (deferred Triton loads) — warm and discard; --reasoning-parser / tool parsers are part of the contract (vLLM Boot Forensics and Config Interrogation).

7. systemd handoff — THE TRAP

./scripts/deploy.sh <host> --service <svc> gates health, but a leftover manual server holding the port makes the health check pass while the UNIT crash-loops on bind. Always confirm ownership: systemctl show <svc> -p MainPID equals the single pgrep -x llama-server (or the container runtime PID); only then trust "healthy". If a real long-context task is queued, do the handoff anyway — client aborts are recoverable, drift is not.

8. Close-out (mandatory)

1. Docs: active runbook updated (build recipe, profile, validation numbers incl. real-workload prefill/decode); superseded runbook gets a RETIRED banner and stays as the rollback record; LLM/index.md mermaid + active list; AGENTS.md fleet table + runtime section + history chain; work-log entry in serverAppConfigs-docs/log.md.
2. ~/.omp/agent/models.yml: contextWindow must equal the REAL per-slot context — llama.cpp logs n_ctx_slot per prompt (--parallel N divides the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports max-model-len and its concurrency line. Overselling causes silent client-side overflow or a discovered-wall 400. Also maxTokens vs the engine's cap, input capabilities (text/image), and reasoningContentField per runtime (llama.cpp/SGLang emit reasoning_content, vLLM emits reasoning). Fix every modelRoles entry that named the old endpoint.
3. Rollback = one env commit (paths + LLAMA_BIN + flags), prior binary + payload kept on disk, rollback numbers recorded in the runbook.

9. Revert-pin an ad hoc experiment (the three pins)

A campaign is "done" only when all three exist — without them a podman pull, an image update, or a lost scratch script destroys the way back:

1. Git tag (annotated, pushed): git tag -a <exp>-tuned-YYYYMMDD -m "<tuned env/flags + measured headline + runbook path + revert command>" — a future agent must reconstruct the campaign from the tag message alone.
2. Image digest in the serve script: floating tags move. Capture podman inspect <ctr> --format '{{.ImageName}} {{.Image}}' + podman images --digests, patch IMAGE= to repo@sha256:<repodigest> with a comment (image ID, date, old ref), verify podman image exists repo@sha256:<digest> and bash -n. Digest refs survive upstream pushes; podman pull repo@sha256:<digest> re-fetches exact bytes. One host can hold MULTIPLE images of one repo — pin by the digest of the RUNNING container, never by tag name. Never edit the pinned line back to a floating tag; new experiments branch from the digest.
3. Local alias: podman tag <id> <exp>-runtime:tuned-YYYYMMDD.

Record scratch-script paths + full flag surface in the runbook (hook env vars in CAPS); expired images/scripts go to .AGENTTRASH, never rm. Pure measurement sessions that changed no state need only pin 1.

10. Research verification discipline (for "can model X run here?")

• Scouted research summaries have shipped fabricated citations on this fleet (invented HF repos, plausible-but-wrong PR numbers, stale engine versions). NEVER copy a scout's citations into a deliverable — re-verify each load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it enters anything.
• The in-repo runbooks are FIRST-HAND evidence and outrank any web claim about our own hosts.
• Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active, CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path (heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K). GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind -c, the default hard-hung a 128 GB host; native FP8 328.3 GB; -fa off MANDATORY, flash-attn breaks MLA correctness; llama.cpp arch glm5next lived in unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp MMLU-Pro vs Q8_0).
• GGUF parsing gotcha: expert tensors are blk.N.ffn_{gate,up,down}_exps.weight — 4 dot-parts INCLUDING .weight; a len(parts) == 3 filter matches zero layers with no parse error. Filter on parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2].

11. Adding a second LLM endpoint to a host

When a host must serve two models at once (an iGPU runtime plus a second card), or one endpoint is moving and a new one takes the freed hardware.

0. Decide the shape before touching hardware. One provider per HOST:PORT in ~/.omp/agent/models.yml (on the box where the harness runs) — a second endpoint is a new provider block, not a second model in the existing one. Read the budget as an inequality: for unified memory the engine's own fit check is the arbiter (budget = host − pinned weights − reserve; need = KV + arena + vision); for a discrete card, weight the KV by counting only full-attention layers — and remember --parallel N DIVIDES --ctx-size. A co-tenant is a budget line, not a footnote: free RAM by changing the POOL or the ARENA (the reserve is a fit-check term, never an allocation) and drop optional charges (a vision tower is ~2 GiB) rather than hoping.

1. Repo first, and retire in the same commit. New service files (scripts/<svc>.env, .service, serve-<svc>.sh with hard guards for missing artifacts); planner entries (SERVICE_ORDER + the target's service set + path mappings, inventories in deploy.sh/deploy-remote.sh, the dispatch case); retire the obsoleted pieces into .AGENTTRASH/ with a suffix — and check what pointed at them: bench harnesses, a chat template only the old runtime read, and scripts/test-changed.sh mappings are the usual victims (a mapping to a retired test file fails the pre-commit wrapper silently for weeks). One bats file per service asserting INVARIANTS (pool == 2 × ctx, KV bytes ≤ card budget, port separation), not constants; a negative grep must strip comments first (grep -vE '^\s*#' file | grep -q …) or it matches the prose explaining the very thing you removed.

2. The port-bind race (rootless podman). pasta binds the published port at container CREATE; a just-stopped predecessor still holds it. Order the handler: retire the old manager's unit → stop THIS service's own unit (no-op on first cutover) → wait for every socket on the port to clear (Container Host Reachability) → install → start. Without the second step a redeploy deadlocks against its own gate.

3. Wire the surfaces, refuse impossible signals. Point the harness small/fast + image roles at the new endpoint; move the vision role WITH the vision tower (one owner per capability). shared/homepage/services.json: a card per endpoint, probed from the monitor's own host. Prometheus: only add a job if the runtime actually exposes Prometheus text (halogen publishes /health, /cache, /v1/* and nothing else — the job was REMOVED with the reason written down rather than left as a permanent red).

4. Cutover, then verify by driving it. deploy.sh <target> --service <large> --service <small> (order matters); per endpoint: /health, /v1/models reports the alias, a chat completion returns content AND reasoning_content, a tools request returns tool_calls, an image request is answered (or refused with a named reason), the runtime's own speculation counters move, and the unit is enabled/generated with linger on. Read the engine's boot ledger for a silent clamp. Finally spend one real request through the harness per changed role — routing labels lie; the servers' own journals do not.

Traps that look like model faults: 500 tools param requires --jinja flag (runtime lacks the Jinja path — pin it); 500 Jinja Exception: Unexpected reasoning effort <level> (fold via compat.reasoningEffortMap); the same request works by hand but a subagent dies (subagents run at a different thinking level — check both); the container dies instantly although the predecessor "stopped" (the port race above).

Related

• GGUF Compatibility — which runtime can load a payload at all
• Llama Cpp Serving Sizing — pricing the shape once it loads
• RDNA4 MoE Expert Offload VRAM Budget — vLLM-side fit arithmetic
• LLM Bench Windows and Measurement — measuring the new deployment
• dankserver-qwen38-fp8-expert-offload-tuning — the current endpoint