---
name: llamacpp-endpoint-model-swap
description: "Swap the model on a dank-fleet llama.cpp endpoint (frank-secondary pattern): fetch, env edit, spec flags, deploy through git, rewrite the bats seam, verify VRAM with the engine's own numbers."
---

# llama.cpp endpoint model swap (dank fleet)

Use when replacing the model/shape on a llama.cpp endpoint (e.g. frank-secondary on the RTX 3090). The engine's boot numbers are the verdict — do NOT commit to a feasibility verdict from head math; boot it and read `nvidia-smi`.

## Sequence

1. **Geometry first, from the loaded model's own log** (`--log-verbosity 4`, or the HF `config.json` under `text_config` for multimodal): layers, `n_head_kv` (an ARRAY — per-layer, e.g. gemma4 has 8 on sliding layers, 2 on full-attention), head_dim, `sliding_window_pattern`, `n_swa`, `attention_k_eq_v`. KV scales only with full-attention layers; sliding layers cost a fixed window-size ring. Single-value HF fields lie about per-layer heterogeneity.
2. **Fit the KV rate law to the engine, not to arithmetic.** Run once with `--log-verbosity 4` and read the `llama_kv_cache` / `sched_reserve` buffer MiB lines. Known law for master 5fc4f3c: bytes/elem = (K bytes + V bytes) **x2** when the file declares `attention_k_eq_v` (6.125 at f16 K + q8_0 V, 2.125 at q8_0/q8_0) — k_eq_v is a compute-graph mapping and does NOT shrink the cache. Full cache = n_full x n_head_kv x head_dim x rate x ctx; SWA cache = n_swa x window x n_head_kv_swa x head_dim x rate. Both reproduce the printed MiB exactly; if your formula does not, your law is wrong, not the engine.
3. **Probe the binary's real capabilities** before promising flags: `--help | grep -A2 cache-type-k` (this fleet's build has NO fp8; q8_0 is the "FP8 class"), `--help | grep -A2 verbosity` (`-lv, --log-verbosity N`; ship `VERBOSITY` as an env tunable + `--log-verbosity "$VERBOSITY"` so VRAM forensics land in the journal), `--help | grep model-draft` for separate drafters. NEVER `grep -c <archname>` on the packed binary to check arch support — it returns 0 for every arch; check the source tree at the build's commit.
4. **Fetch with `-f` + byte-count check** into `/home/dank/models/<name>/`.
5. **Edit the repo trio** (`scripts/serve-<svc>.sh`, `scripts/<svc>.env`, `~/.omp/agent/models.yml`): spec flags move as a SET (`SPEC_TYPE=none` strips `--spec-type` AND `--spec-draft-model`/`--spec-draft-*`); a drafter with its own arch file (gemma4-assistant) needs `SPEC_DRAFT_FILE`, and its draft KV is built against the TARGET's full window — price it from a boot pair. `models.yml` `contextWindow` = per-slot window (`CTX_SIZE/PARALLEL`); recompute `maxTokens` for the compaction clamp `min(maxTokens, ctx - estimate - 4096)`.
6. **Rewrite the bats seam in the same commit** (`tests/<svc>_unit.bats`): payload path, slot geometry, KV pair, spec state, batch pair, verbosity, mmproj opt-out, reasoning flags — all argv-asserted against the stub engine; where the contract is a repo value (batch pair, verbosity) assert argv == env value so a retune moves one file.
7. **Deploy through git** (no direct remote edits). If the remote tree is dirty: `git stash push <files>` on the host, `git pull --ff-only`, land the change as a commit. A pull that aborts leaves the host at the OLD revision while the service runs the NEW edits — verify `git log -1` on the host.
8. **Verify the engine, not your math**: `/proc/<pid>/cmdline` (argv is the contract), `/v1/models` (`n_ctx`, `ftype`, `n_ctx_train`), `nvidia-smi --query-gpu=memory.used,memory.free`, `/health` 200, a vision probe (solid color >=256 px via `image_url`, expect the exact word), and one generation probe reading `timings.predicted_per_second`. Record the numbers in the env comment + runbook, then push and ff-pull the host.
9. **Cutover AGENTS.md in the same commit**: fleet-topology table row, the `### <svc>` section, the `Model Config` contextWindow/vision note, the runbook index ACTIVE row, a RETIRED row for the superseded era. The new docs page must be linked from the TOP-LEVEL `serverAppConfigs-docs/index.md`.

## Gotchas

- A systemd restart reporting `inactive` in the first seconds is usually model-load time; check `journalctl -u <svc>` for `model done`/`server is listening` before declaring failure.
- Rollback = the previous payload stays on disk; one `MODEL_FILE` + shape edit back.
- A thinking model can consume the whole token budget in `reasoning_content` (empty `content`, `finish_reason: length`) — probe with a budget that clears thinking.
- **Bats vacuity trap**: `serve_argv` REWRITES `$STAGE/test.env` from the shipped env, so a test that mutates the file (`sed -i`, append) then calls `serve_argv` asserts on regenerated content — a tautology. Tests that mutate must exec via a `run_stage` helper running the file as-is. Symptom: the assertion fails only after you fix the vacuity, and the env dump shows the mutation gone.
- `env NAME=value cmd` needs valid identifiers; a flag-like token (`nvidia-modprobe=true`) is only legal as an `env` ARGUMENT, not as a shell assignment prefix.