live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/llamacpp-endpoint-model-swap/SKILL.md

@@ -1,28 +1,33 @@
 ---
 name: llamacpp-endpoint-model-swap
-description: "Swap the model on a dank-fleet llama.cpp endpoint (frank-secondary pattern): fetch, env edit, spec flags, deploy through git, rewrite the bats seam, verify VRAM with the engine's own numbers."
+description: "Swap the model on a dank-fleet llama.cpp endpoint: fetch, runtime-door census, env edit, deploy through git, rewrite the bats seam, sweep every live-path straggler, verify VRAM with the engine's own numbers."
 ---
 
 # llama.cpp endpoint model swap (dank fleet)
 
-Use when replacing the model/shape on a llama.cpp endpoint (e.g. frank-secondary on the RTX 3090). The engine's boot numbers are the verdict — do NOT commit to a feasibility verdict from head math; boot it and read `nvidia-smi`.
+Use when replacing the model/shape on a llama.cpp endpoint (frank-zealot on the RTX 3090). The engine's boot numbers are the verdict — do NOT commit to a feasibility verdict from head math; boot it and read `nvidia-smi`.
 
 ## Sequence
 
-1. **Geometry first, from the loaded model's own log** (`--log-verbosity 4`, or the HF `config.json` under `text_config` for multimodal): layers, `n_head_kv` (an ARRAY — per-layer, e.g. gemma4 has 8 on sliding layers, 2 on full-attention), head_dim, `sliding_window_pattern`, `n_swa`, `attention_k_eq_v`. KV scales only with full-attention layers; sliding layers cost a fixed window-size ring. Single-value HF fields lie about per-layer heterogeneity.
-2. **Fit the KV rate law to the engine, not to arithmetic.** Run once with `--log-verbosity 4` and read the `llama_kv_cache` / `sched_reserve` buffer MiB lines. Known law for master 5fc4f3c: bytes/elem = (K bytes + V bytes) **x2** when the file declares `attention_k_eq_v` (6.125 at f16 K + q8_0 V, 2.125 at q8_0/q8_0) — k_eq_v is a compute-graph mapping and does NOT shrink the cache. Full cache = n_full x n_head_kv x head_dim x rate x ctx; SWA cache = n_swa x window x n_head_kv_swa x head_dim x rate. Both reproduce the printed MiB exactly; if your formula does not, your law is wrong, not the engine.
-3. **Probe the binary's real capabilities** before promising flags: `--help | grep -A2 cache-type-k` (this fleet's build has NO fp8; q8_0 is the "FP8 class"), `--help | grep -A2 verbosity` (`-lv, --log-verbosity N`; ship `VERBOSITY` as an env tunable + `--log-verbosity "$VERBOSITY"` so VRAM forensics land in the journal), `--help | grep model-draft` for separate drafters. NEVER `grep -c <archname>` on the packed binary to check arch support — it returns 0 for every arch; check the source tree at the build's commit.
-4. **Fetch with `-f` + byte-count check** into `/home/dank/models/<name>/`.
-5. **Edit the repo trio** (`scripts/serve-<svc>.sh`, `scripts/<svc>.env`, `~/.omp/agent/models.yml`): spec flags move as a SET (`SPEC_TYPE=none` strips `--spec-type` AND `--spec-draft-model`/`--spec-draft-*`); a drafter with its own arch file (gemma4-assistant) needs `SPEC_DRAFT_FILE`, and its draft KV is built against the TARGET's full window — price it from a boot pair. `models.yml` `contextWindow` = per-slot window (`CTX_SIZE/PARALLEL`); recompute `maxTokens` for the compaction clamp `min(maxTokens, ctx - estimate - 4096)`.
-6. **Rewrite the bats seam in the same commit** (`tests/<svc>_unit.bats`): payload path, slot geometry, KV pair, spec state, batch pair, verbosity, mmproj opt-out, reasoning flags — all argv-asserted against the stub engine; where the contract is a repo value (batch pair, verbosity) assert argv == env value so a retune moves one file.
-7. **Deploy through git** (no direct remote edits). If the remote tree is dirty: `git stash push <files>` on the host, `git pull --ff-only`, land the change as a commit. A pull that aborts leaves the host at the OLD revision while the service runs the NEW edits — verify `git log -1` on the host.
-8. **Verify the engine, not your math**: `/proc/<pid>/cmdline` (argv is the contract), `/v1/models` (`n_ctx`, `ftype`, `n_ctx_train`), `nvidia-smi --query-gpu=memory.used,memory.free`, `/health` 200, a vision probe (solid color >=256 px via `image_url`, expect the exact word), and one generation probe reading `timings.predicted_per_second`. Record the numbers in the env comment + runbook, then push and ff-pull the host.
-9. **Cutover AGENTS.md in the same commit**: fleet-topology table row, the `### <svc>` section, the `Model Config` contextWindow/vision note, the runbook index ACTIVE row, a RETIRED row for the superseded era. The new docs page must be linked from the TOP-LEVEL `serverAppConfigs-docs/index.md`.
+1. **Runtime door BEFORE anything else.** Histogram the GGUF's tensor types and metadata (one python range-read of the KV section): a custom type (e.g. PQ2_0 = GGML type 142 + `prism.hadamard.*`) means the vendor/fork build is FORCED, not chosen — stock llama.cpp refuses it at load. Record the build identity (path, build number, commit, sha256) in the env comment; a `--help` census on that binary is what settles flag questions (this fleet's builds have NO fp8 KV; q8_0 is the "FP8 class"). NEVER `grep -c <archname>` on the packed binary to check arch support — it returns 0 for every arch.
+2. **Geometry first, from the loaded model's own log** (`--log-verbosity 4`, or the HF `config.json` under `text_config` for multimodal): layers, `n_head_kv` (an ARRAY — per-layer), head_dim, `sliding_window_pattern`, `n_swa`, `attention_k_eq_v`. KV scales only with full-attention layers; sliding layers cost a fixed ring; recurrent/linear layers (Gated DeltaNet) bill a fixed state per SLOT (~150 MiB measured), invisible to pool arithmetic. Single-value HF fields lie about per-layer heterogeneity.
+3. **Fit the KV rate law to the engine, not to arithmetic.** One boot at `-lv 4` prints `llama_kv_cache` / `sched_reserve` MiB lines. Master 5fc4f3c law: bytes/elem = (K + V) **x2** when the file declares `attention_k_eq_v`. Full cache = n_full x n_head_kv x head_dim x rate x ctx. If your formula disagrees with the printed MiB, your law is wrong.
+4. **Price the KV dtype by BOOTING the arms, at the target window.** f16 may simply OOM (`cudaMalloc failed` after the engine's own fit line names the MiB it wanted); q8_0 may be the only pair that funds 2 x 200,704; q4_0 is the fallback rung. Record every arm's used/free. **The mmproj lands AFTER the engine's fit projection** (projected fit < settled VRAM by ~1.5x the projector file size) — quote both numbers.
+5. **Fetch with `-f` + byte-count + sha256 check** into `/home/dank/models/<name>/`; `--verify-only` mode for re-census.
+6. **Edit the repo trio** (`scripts/serve-<svc>.sh`, `scripts/<svc>.env`, `~/.omp/agent/models.yml`). **Use the serve script's OWN variable names** (`CACHE_TYPE_K`/`CACHE_TYPE_V`, `BIND_ADDR`, `NGL`) — an invented name is silently swallowed by `: "${VAR:=default}"` and the arm boots the DEFAULT (cost a whole trial window: a `KV_CACHE_TYPE` scratch env booted f16 and OOMed while reading as a model fault). Sweep the script's DEFAULTS to the new payload too: a defaulted run must land on the shipped shape, never the retired one. Spec flags move as a SET (`SPEC_TYPE=none` strips `--spec-type` AND `--spec-draft-*`); MTP is a property of the FILE (no `nextn` tensors = no drafter, ever). `models.yml` `contextWindow` = per-slot window (`CTX_SIZE/PARALLEL`); recompute `maxTokens` for the compaction clamp `min(maxTokens, ctx - estimate - 4096)`. Check the payload's own template: if it reads `reasoning_effort`, the `reasoningEffortMap` fold becomes LOAD-BEARING (an unmapped level is a 500 naming Jinja, not the model) and quote the file's `general.sampling.*` trio in `extraBody`.
+7. **Rewrite the bats seam in the same commit** (`tests/<svc>_unit.bats`): payload trio paths, slot geometry, spec state, batch pair, verbosity, mmproj opt-out, reasoning flags — argv-asserted against the stub engine; where the contract is a repo value (batch pair, verbosity, **KV dtype**) assert argv == env value so a retune moves one file and a wrong-name env edit cannot pass silently.
+8. **Trial window protocol** (when the shape is unproven on the card): stop the unit (trap-guarded `systemctl start` restore in the scratch script), run the arm census + behavior probes as a DETACHED script writing to `/home/dank/logs/<name>.out`, then before touching production CENSUS the live state: `systemctl is-active`, `ss -ltnp | grep <port>`, `pgrep -x llama-server`, `nvidia-smi`. A scratch server orphaned by a client-side timeout holds the whole card and will OOM the real boot — kill by PID (never `pkill -f llama-server`). A trap restore can race your manual kill; read the script's own `RESTORE` log line before assuming who did what.
+9. **Deploy through git** (no direct remote edits): commit atomically on the worktree branch, `git fetch && git rebase origin/main` (log.md conflicts at EOF are two waves appending — keep both), push branch (`--force-with-lease` after a rebase), then in the MAIN checkout `git pull --ff-only` + `git merge --ff-only <branch>` + `git push origin main`, then `scripts/deploy.sh <host> --service <svc>` (`--service` is REPEATABLE — name the homepage too when cards changed, or the target ships the model without its docs). A `deploy.sh` with no `--service` is a fleet-wide drain: never.
+10. **Verify the engine, not your math**: `systemctl is-active/is-enabled` + the new `Description`, the journal fit/KV/buffer ledger (`projected to use N`, `KV buffer size`, `n_ctx_seq`, `n_slots`), `/v1/models` served alias (radiance-style echo does not apply to llama.cpp, but the alias is still the honest check), `nvidia-smi` settled, then live probes through the REAL request shape the harness sends (exact-instruction chat + `reasoning_content`, `tool_calls` JSON args, solid-color `image_url` >=256 px expecting the exact word, `timings.predicted_per_second`). Never report the FIRST request after a boot (warmup artifact). Push, and record numbers in the env comment + runbook.
+11. **Cutover every consumer, same branch**: AGENTS.md (topology row, `### <svc>` section, Model Config note, runbook index ACTIVE + RETIRED rows); the new docs page MUST be linked from the TOP-LEVEL `serverAppConfigs-docs/index.md` (the `docs_completeness` seam fails otherwise, not from `LLM/index.md`); `shared/homepage/services.json` + `docs.html` (model link, spec line, rate, and the WARNING — a censored base breaks the "common uncensored text" equality in `homepage_config.bats`, so re-key the test's premise too); `serverAppConfigs-docs/Dankclaw/OpenWebUI.md` + the live `model` row `meta.description` (live DB state: snapshot `webui.db.bak-<date><letter>`, UPDATE via host python3 over ssh; do NOT restart `open-webui` for a cosmetic string — it logs every family session out).
 
 ## Gotchas
 
+- A "we can rebuild the fork ourselves for provenance" claim needs the source tree AT the vendor build's commit — check `git -C <src> log --oneline -1` first. A fork checkout at branch HEAD (`prism` at `0082ac2`) is NOT the vendor build (`2459f68b5`), so the rebuild produces a different binary, not the verified one. Also: if the vendor binary already demonstrates the requirement (KV dtype buffer math in the ledger + live probes), a rebuild buys no behavior change — offer it explicitly as provenance hardening, never do it silently.
 - A systemd restart reporting `inactive` in the first seconds is usually model-load time; check `journalctl -u <svc>` for `model done`/`server is listening` before declaring failure.
-- Rollback = the previous payload stays on disk; one `MODEL_FILE` + shape edit back.
-- A thinking model can consume the whole token budget in `reasoning_content` (empty `content`, `finish_reason: length`) — probe with a budget that clears thinking.
-- **Bats vacuity trap**: `serve_argv` REWRITES `$STAGE/test.env` from the shipped env, so a test that mutates the file (`sed -i`, append) then calls `serve_argv` asserts on regenerated content — a tautology. Tests that mutate must exec via a `run_stage` helper running the file as-is. Symptom: the assertion fails only after you fix the vacuity, and the env dump shows the mutation gone.
-- `env NAME=value cmd` needs valid identifiers; a flag-like token (`nvidia-modprobe=true`) is only legal as an `env` ARGUMENT, not as a shell assignment prefix.
+- Rollback = the previous payload stays on disk; one `MODEL_FILE` + shape edit back. Record a refusal-freedom fallback the same way (an official censored base ↔ a CRACK re-export with byte-identical header geometry is a one-line swap; prove the identity with the tensor histogram, not the card).
+- A thinking model can consume the whole token budget in `reasoning_content` (empty `content`, `finish_reason: length`) — probe with a budget that clears thinking; a short-budget vision probe reads as "the model is blind".
+- `/completion` at `-lv 4` prints `failed to parse reasoning content ...` on raw prompts closing with an end-of-turn marker: cosmetic (chat path parses correctly). At ~26k+ a bare `/completion` can return 0 tokens with no error — raw context has no role frame; chat is unaffected.
+- **Bats vacuity trap**: `serve_argv` REWRITES `$STAGE/test.env` from the shipped env, so a test that mutates the file then calls `serve_argv` asserts on regenerated content — a tautology. Tests that mutate must exec via a `run_stage` helper running the file as-is.
+- `env NAME=value cmd` needs valid identifiers; a flag-like token is only legal as an `env` ARGUMENT, not as a shell assignment prefix.
+- Gitea `git push` on a rebased branch is rejected non-fast-forward: `--force-with-lease` (your own branch), never on main.