command-center/agent/managed-skills/llamacpp-endpoint-model-swap/SKILL.md

name, description

name	description
llamacpp-endpoint-model-swap	Swap the model on a dank-fleet llama.cpp endpoint (frank-secondary pattern): fetch, env edit, spec flags, deploy through git, rewrite the bats seam, verify VRAM with the engine's own numbers.

llama.cpp endpoint model swap (dank fleet)

Use when replacing the model/shape on a llama.cpp endpoint (e.g. frank-secondary on the RTX 3090). The engine's boot numbers are the verdict — do NOT commit to a feasibility verdict from head math; boot it and read nvidia-smi.

Sequence

1. Geometry first, from the loaded model's own log (--log-verbosity 4, or the HF config.json under text_config for multimodal): layers, n_head_kv (an ARRAY — per-layer, e.g. gemma4 has 8 on sliding layers, 2 on full-attention), head_dim, sliding_window_pattern, n_swa, attention_k_eq_v. KV scales only with full-attention layers; sliding layers cost a fixed window-size ring. Single-value HF fields lie about per-layer heterogeneity.
2. Fit the KV rate law to the engine, not to arithmetic. Run once with --log-verbosity 4 and read the llama_kv_cache / sched_reserve buffer MiB lines. Known law for master 5fc4f3c: bytes/elem = (K bytes + V bytes) x2 when the file declares attention_k_eq_v (6.125 at f16 K + q8_0 V, 2.125 at q8_0/q8_0) — k_eq_v is a compute-graph mapping and does NOT shrink the cache. Full cache = n_full x n_head_kv x head_dim x rate x ctx; SWA cache = n_swa x window x n_head_kv_swa x head_dim x rate. Both reproduce the printed MiB exactly; if your formula does not, your law is wrong, not the engine.
3. Probe the binary's real capabilities before promising flags: --help | grep -A2 cache-type-k (this fleet's build has NO fp8; q8_0 is the "FP8 class"), --help | grep -A2 verbosity (-lv, --log-verbosity N; ship VERBOSITY as an env tunable + --log-verbosity "$VERBOSITY" so VRAM forensics land in the journal), --help | grep model-draft for separate drafters. NEVER grep -c <archname> on the packed binary to check arch support — it returns 0 for every arch; check the source tree at the build's commit.
4. Fetch with -f + byte-count check into /home/dank/models/<name>/.
5. Edit the repo trio (scripts/serve-<svc>.sh, scripts/<svc>.env, ~/.omp/agent/models.yml): spec flags move as a SET (SPEC_TYPE=none strips --spec-type AND --spec-draft-model/--spec-draft-*); a drafter with its own arch file (gemma4-assistant) needs SPEC_DRAFT_FILE, and its draft KV is built against the TARGET's full window — price it from a boot pair. models.yml contextWindow = per-slot window (CTX_SIZE/PARALLEL); recompute maxTokens for the compaction clamp min(maxTokens, ctx - estimate - 4096).
6. Rewrite the bats seam in the same commit (tests/<svc>_unit.bats): payload path, slot geometry, KV pair, spec state, batch pair, verbosity, mmproj opt-out, reasoning flags — all argv-asserted against the stub engine; where the contract is a repo value (batch pair, verbosity) assert argv == env value so a retune moves one file.
7. Deploy through git (no direct remote edits). If the remote tree is dirty: git stash push <files> on the host, git pull --ff-only, land the change as a commit. A pull that aborts leaves the host at the OLD revision while the service runs the NEW edits — verify git log -1 on the host.
8. Verify the engine, not your math: /proc/<pid>/cmdline (argv is the contract), /v1/models (n_ctx, ftype, n_ctx_train), nvidia-smi --query-gpu=memory.used,memory.free, /health 200, a vision probe (solid color >=256 px via image_url, expect the exact word), and one generation probe reading timings.predicted_per_second. Record the numbers in the env comment + runbook, then push and ff-pull the host.
9. Cutover AGENTS.md in the same commit: fleet-topology table row, the ### <svc> section, the Model Config contextWindow/vision note, the runbook index ACTIVE row, a RETIRED row for the superseded era. The new docs page must be linked from the TOP-LEVEL serverAppConfigs-docs/index.md.

Gotchas

• A systemd restart reporting inactive in the first seconds is usually model-load time; check journalctl -u <svc> for model done/server is listening before declaring failure.
• Rollback = the previous payload stays on disk; one MODEL_FILE + shape edit back.
• A thinking model can consume the whole token budget in reasoning_content (empty content, finish_reason: length) — probe with a budget that clears thinking.
• Bats vacuity trap: serve_argv REWRITES $STAGE/test.env from the shipped env, so a test that mutates the file (sed -i, append) then calls serve_argv asserts on regenerated content — a tautology. Tests that mutate must exec via a run_stage helper running the file as-is. Symptom: the assertion fails only after you fix the vacuity, and the env dump shows the mutation gone.
• env NAME=value cmd needs valid identifiers; a flag-like token (nvidia-modprobe=true) is only legal as an env ARGUMENT, not as a shell assignment prefix.