Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
42 lines
3.9 KiB
Markdown
42 lines
3.9 KiB
Markdown
---
|
|
name: llm-fleet-model-swap
|
|
description: "Swap the model/runtime behind frank-default-large/small on Dankclaw/Dankserver via the git flow, with pre-deploy live validation and the port-holder health-check trap check."
|
|
---
|
|
|
|
# LLM fleet model swap (frank-default-large / frank-default-small)
|
|
|
|
Use when switching model, quant, or llama.cpp/FreeToken runtime for a `frank-default-*` service in `~/Projects/serverAppConfigs`.
|
|
|
|
## 0. Ground truth first
|
|
- Read the HF model card (runtime pin, hard constraints, recommended args) + repo files `scripts/frank-default-<size>.env`, `scripts/serve-frank-<size>.sh`, `scripts/frank-default-<size>.service`, and the runbook under `serverAppConfigs-docs/LLM/`.
|
|
- Check the host before downloading anything: `ls /home/dank/models/`, `df -h /home`, `du -sh` — bench-era or prior-deployment files often already exist.
|
|
|
|
## 1. Artifact integrity
|
|
- List repo files: `curl -s https://huggingface.co/api/models/<repo>/tree/main?recursive=true` → `.lfs.oid` holds the sha256.
|
|
- On-host file: compare size AND `sha256sum` against the LFS oid before trusting any existing copy (hash runs ~few min for >100 GiB; run it in background).
|
|
- New download on Dankserver: `/home/dank/freetoken/.venv/bin/hf download <repo> --local-dir ...`.
|
|
|
|
## 2. Runtime (if the card pins a fork/build)
|
|
- Clone + build with the card's exact cmake line, **plus `-DLLAMA_BUILD_WEBUI=OFF`** — WebUI asset provisioning (npm / HF llama-ui bucket) aborts llama.cpp link at 98% on these hosts; API server unaffected.
|
|
- Verify every flag our serve script passes exists in the new binary before touching the service:
|
|
`llama-server --help | grep -q -- "<flag>"` for each; list MISSes.
|
|
|
|
## 3. Config via git only
|
|
- Edit `scripts/frank-default-<size>.env` (LLAMA_BIN, MODEL_DIR, MODEL_FILE, MMPROJ_PATH) + serve script header/defaults; `bash -n` the script; `./tests/run.sh` must pass; commit; push; `git pull --ff-only` on the target.
|
|
- Do NOT edit configs on the remote. Renaming a model dir (`mv`) is host state, allowed.
|
|
|
|
## 4. Live validation (no scratch ports — GPU/RAM can't host two big models)
|
|
- Ensure the old unit is truly gone: `systemctl is-active` may hang in `stop-sigterm` because llama.cpp never preempts an in-flight generation; poll until the PID exits before launching.
|
|
- Stop service, run the real serve script manually (`setsid nohup bash scripts/serve-frank-<size>.sh > /tmp/x.log 2>&1 &` — plain `ssh 'nohup ... &'` dies with its ssh when foregrounded; keep ssh under timeout or it gets killed mid-stop).
|
|
- Verify: health 200; a real `/v1/chat/completions` (FreeToken REQUIRES explicit `"model"` field; llama.cpp servers don't care); reasoning flows (`reasoning_content` non-empty; content may be empty with finish=length if the thinking eats the budget).
|
|
|
|
## 5. systemd handoff + THE TRAP
|
|
- `./scripts/deploy.sh <host> --service <svc>` — but a leftover manual server holding :7770 makes the health check pass while the UNIT crash-loops on bind.
|
|
- Always confirm ownership: `systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server` (or the ft serve PID); only then trust "healthy after Ns".
|
|
- If a real long-context task is queued/running, do the handoff anyway (client aborts are recoverable; drift is not) — or defer cosmetic-only redeploys.
|
|
|
|
## 6. Close-out (mandatory)
|
|
- Docs: active runbook updated (build recipe, profile, validation numbers incl. real-workload prefill/decode), superseded runbook gets RETIRED banner kept as rollback record, `LLM/index.md` mermaid + active list, AGENTS.md fleet table + Runtime section + history chain, work log entry in `serverAppConfigs-docs/log.md`.
|
|
- `~/.pi/agent/models.json`: contextWindow must equal the REAL per-slot context — llama.cpp logs `n_ctx_slot` per prompt; parallel N clamps `n_ctx_train` (e.g. 262144/2 = 131072/slot). Overselling causes silent client-side overflow.
|
|
- Rollback = one env commit (paths + LLAMA_BIN); keep prior binary + payload on disk.
|