command-center/agent/managed-skills/llm-fleet-model-swap/SKILL.md

name, description

name	description
llm-fleet-model-swap	Swap the model/runtime behind frank-default-large/small on Dankclaw/Dankserver via the git flow, with pre-deploy live validation and the port-holder health-check trap check.

LLM fleet model swap (frank-default-large / frank-default-small)

Use when switching model, quant, or llama.cpp/FreeToken runtime for a frank-default-* service in ~/Projects/serverAppConfigs.

0. Ground truth first

• Read the HF model card (runtime pin, hard constraints, recommended args) + repo files scripts/frank-default-<size>.env, scripts/serve-frank-<size>.sh, scripts/frank-default-<size>.service, and the runbook under serverAppConfigs-docs/LLM/.
• Check the host before downloading anything: ls /home/dank/models/, df -h /home, du -sh — bench-era or prior-deployment files often already exist.

1. Artifact integrity

• List repo files: curl -s https://huggingface.co/api/models/<repo>/tree/main?recursive=true → .lfs.oid holds the sha256.
• On-host file: compare size AND sha256sum against the LFS oid before trusting any existing copy (hash runs ~few min for >100 GiB; run it in background).
• New download on Dankserver: /home/dank/freetoken/.venv/bin/hf download <repo> --local-dir ....

2. Runtime (if the card pins a fork/build)

• Clone + build with the card's exact cmake line, plus -DLLAMA_BUILD_WEBUI=OFF — WebUI asset provisioning (npm / HF llama-ui bucket) aborts llama.cpp link at 98% on these hosts; API server unaffected.
• Verify every flag our serve script passes exists in the new binary before touching the service: llama-server --help | grep -q -- "<flag>" for each; list MISSes.

3. Config via git only

• Edit scripts/frank-default-<size>.env (LLAMA_BIN, MODEL_DIR, MODEL_FILE, MMPROJ_PATH) + serve script header/defaults; bash -n the script; ./tests/run.sh must pass; commit; push; git pull --ff-only on the target.
• Do NOT edit configs on the remote. Renaming a model dir (mv) is host state, allowed.

4. Live validation (no scratch ports — GPU/RAM can't host two big models)

• Ensure the old unit is truly gone: systemctl is-active may hang in stop-sigterm because llama.cpp never preempts an in-flight generation; poll until the PID exits before launching.
• Stop service, run the real serve script manually (setsid nohup bash scripts/serve-frank-<size>.sh > /tmp/x.log 2>&1 & — plain ssh 'nohup ... &' dies with its ssh when foregrounded; keep ssh under timeout or it gets killed mid-stop).
• Verify: health 200; a real /v1/chat/completions (FreeToken REQUIRES explicit "model" field; llama.cpp servers don't care); reasoning flows (reasoning_content non-empty; content may be empty with finish=length if the thinking eats the budget).

5. systemd handoff + THE TRAP

• ./scripts/deploy.sh <host> --service <svc> — but a leftover manual server holding :7770 makes the health check pass while the UNIT crash-loops on bind.
• Always confirm ownership: systemctl show <svc> -p MainPID equals the single pgrep -x llama-server (or the ft serve PID); only then trust "healthy after Ns".
• If a real long-context task is queued/running, do the handoff anyway (client aborts are recoverable; drift is not) — or defer cosmetic-only redeploys.

6. Close-out (mandatory)

• Docs: active runbook updated (build recipe, profile, validation numbers incl. real-workload prefill/decode), superseded runbook gets RETIRED banner kept as rollback record, LLM/index.md mermaid + active list, AGENTS.md fleet table + Runtime section + history chain, work log entry in serverAppConfigs-docs/log.md.
• ~/.omp/agent/models.yml: contextWindow must equal the REAL per-slot context — llama.cpp logs n_ctx_slot per prompt; parallel N clamps n_ctx_train (e.g. 262144/2 = 131072/slot). Overselling causes silent client-side overflow.
• Rollback = one env commit (paths + LLAMA_BIN); keep prior binary + payload on disk.