---
name: llamacpp-trial-boot-census
description: "Measure whether a candidate GGUF fits a live llama.cpp endpoint's card: verify serve-script env names first, run a dtype/slot arm matrix with trap-guarded production restore, read the engine fit ledger."
---

# llama.cpp trial-boot census (does candidate X fit this card?)

Paper arithmetic on KV rates is a hypothesis. The engine's own fit ledger plus a
real boot are the proof. This protocol ran on frank-zealot (RTX 3090,
Qwen3.5-9B Q6_K → prism-ml Ternary-Bonsai-2-27B PQ2_0, 2026-10-17).

## 0. Gate: env variable names, not invented ones

A trial env written from memory silently falls back to the serve script's
defaults. `KV_CACHE_TYPE=q8_0` when the script reads `CACHE_TYPE_K`/`CACHE_TYPE_V`
boots **f16** and every later number is wrong (this wasted a whole outage; two
arms printed identical allocations, which is the tell).

```bash
grep -nE ': "\$\{[A-Z_]+[:=-]' scripts/serve-<svc>.sh   # the real names + defaults
grep -nE '\-\-[a-z-]+ "\$' scripts/serve-<svc>.sh       # what actually reaches argv
```

Never invent: KV dtype, bind addr, n-gpu-layers, log-file (units redirect it).

## 1. Runner skeleton

One script on the host, `set -u`, `trap restore EXIT`:

```bash
restore(){ pkill -x llama-server; for i in $(seq 1 90); do [[ "$(used)" -lt 1500 ]] && break; sleep 2; done
  sudo systemctl start <svc>; for i in $(seq 1 150); do [[ "$(curl -s -o /dev/null -w %{http_code} $BASE/health)" == 200 ]] && break; sleep 2; done; }
trap restore EXIT
sudo systemctl stop <svc>            # benching law: stop first, mask fails on relinked units
setsid nohup env FRANK_<SVC>_ENV_FILE=/tmp/armA.env bash scripts/serve-<svc>.sh >$LOG 2>&1 &
```

Launch with `setsid nohup … &`, wait with ONE async watcher (`grep -q DONE`),
never poll. Per-arm drop-in envs via `sed` on a base env — one variable per arm.

## 2. Arm matrix (cheap, each ≈ boot + probes)

| Arm | Purpose |
|---|---|
| target shape × target KV dtype | the verdict + behavior probes |
| target shape × f16 KV | proves the wall (or funds the shape) |
| 1 × native-window | answers "can we serve the full context" |
| target shape × q4_0 | the roomy fallback rung |

## 3. Read the ledger, not `nvidia-smi` alone

```bash
grep -a -A8 "memory breakdown" $LOG     # total = free + (model + context + compute)
grep -aE "KV buffer size|model buffer size|compute buffer size|estimated worst-case memory usage of mmproj|projected to use|cannot meet free memory target|cudaMalloc failed" $LOG
```

Laws learned the hard way:
- `projected to use N MiB vs M MiB of free` + `cannot meet free memory target of R` ⇒ requirement is `projected + R ≤ free`. With `--n-gpu-layers` pinned by the user the fitter **aborts** instead of shrinking — the boot then dies at the KV `cudaMalloc`.
- `context` bills **more than the KV pool**: hybrid models add a fixed recurrent state per slot (~150 MiB/slot for qwen35's 48 DeltaNet layers). Pool math alone under-prices.
- **The projector's `estimated worst-case memory usage of mmproj is X MiB` is NOT inside the fit projection.** It lands after the fit check, so real headroom = paper headroom − X. This is the "boots fine, then dies / dark" class.
- q8_0 K+V bills exactly `2 × (layers_kv × heads × dim / 32 blocks × 34 B)`/token — check the reported KV MiB against that to prove the dtype actually took.

## 4. Probe traps

- **First request after boot is warm-up garbage**: it paid 2.6 t/s prefill / 37.7 t/s decode where the warm run gave 1,258 t/s. Never bench the first request; discard it or warm with a throwaway.
- Reasoning-on chat probes need **≥512 `max_tokens`**: at 64 the answer was empty (`finish=length` mid-`reasoning_content`) while vision actually worked. Judge `content` only at a budget the thinking can fit.
- Raw `/completion` (no chat template) returns `content` only — thinking is invisible; `timings.prompt_n` / `predicted_per_second` are the real keys, not `prompt_tokens`.
- Vision probe: solid-color PNG ≥256 px, expect an exact color word.
- Concurrency honesty: fire two identical requests in parallel; per-slot rates from each response's `timings`.

## 5. Restore proof

`health=200` **and** `systemctl is-active` — an `inactive` unit behind a 200 means
a stray trial process holds the port. Kill by pid; never `pkill -f llama-server`.