75 lines
4.3 KiB
Markdown
75 lines
4.3 KiB
Markdown
---
|
||
name: llamacpp-trial-boot-census
|
||
description: "Measure whether a candidate GGUF fits a live llama.cpp endpoint's card: verify serve-script env names first, run a dtype/slot arm matrix with trap-guarded production restore, read the engine fit ledger."
|
||
---
|
||
|
||
# llama.cpp trial-boot census (does candidate X fit this card?)
|
||
|
||
Paper arithmetic on KV rates is a hypothesis. The engine's own fit ledger plus a
|
||
real boot are the proof. This protocol ran on frank-zealot (RTX 3090,
|
||
Qwen3.5-9B Q6_K → prism-ml Ternary-Bonsai-2-27B PQ2_0, 2026-10-17).
|
||
|
||
## 0. Gate: env variable names, not invented ones
|
||
|
||
A trial env written from memory silently falls back to the serve script's
|
||
defaults. `KV_CACHE_TYPE=q8_0` when the script reads `CACHE_TYPE_K`/`CACHE_TYPE_V`
|
||
boots **f16** and every later number is wrong (this wasted a whole outage; two
|
||
arms printed identical allocations, which is the tell).
|
||
|
||
```bash
|
||
grep -nE ': "\$\{[A-Z_]+[:=-]' scripts/serve-<svc>.sh # the real names + defaults
|
||
grep -nE '\-\-[a-z-]+ "\$' scripts/serve-<svc>.sh # what actually reaches argv
|
||
```
|
||
|
||
Never invent: KV dtype, bind addr, n-gpu-layers, log-file (units redirect it).
|
||
|
||
## 1. Runner skeleton
|
||
|
||
One script on the host, `set -u`, `trap restore EXIT`:
|
||
|
||
```bash
|
||
restore(){ pkill -x llama-server; for i in $(seq 1 90); do [[ "$(used)" -lt 1500 ]] && break; sleep 2; done
|
||
sudo systemctl start <svc>; for i in $(seq 1 150); do [[ "$(curl -s -o /dev/null -w %{http_code} $BASE/health)" == 200 ]] && break; sleep 2; done; }
|
||
trap restore EXIT
|
||
sudo systemctl stop <svc> # benching law: stop first, mask fails on relinked units
|
||
setsid nohup env FRANK_<SVC>_ENV_FILE=/tmp/armA.env bash scripts/serve-<svc>.sh >$LOG 2>&1 &
|
||
```
|
||
|
||
Launch with `setsid nohup … &`, wait with ONE async watcher (`grep -q DONE`),
|
||
never poll. Per-arm drop-in envs via `sed` on a base env — one variable per arm.
|
||
|
||
## 2. Arm matrix (cheap, each ≈ boot + probes)
|
||
|
||
| Arm | Purpose |
|
||
|---|---|
|
||
| target shape × target KV dtype | the verdict + behavior probes |
|
||
| target shape × f16 KV | proves the wall (or funds the shape) |
|
||
| 1 × native-window | answers "can we serve the full context" |
|
||
| target shape × q4_0 | the roomy fallback rung |
|
||
|
||
## 3. Read the ledger, not `nvidia-smi` alone
|
||
|
||
```bash
|
||
grep -a -A8 "memory breakdown" $LOG # total = free + (model + context + compute)
|
||
grep -aE "KV buffer size|model buffer size|compute buffer size|estimated worst-case memory usage of mmproj|projected to use|cannot meet free memory target|cudaMalloc failed" $LOG
|
||
```
|
||
|
||
Laws learned the hard way:
|
||
- `projected to use N MiB vs M MiB of free` + `cannot meet free memory target of R` ⇒ requirement is `projected + R ≤ free`. With `--n-gpu-layers` pinned by the user the fitter **aborts** instead of shrinking — the boot then dies at the KV `cudaMalloc`.
|
||
- `context` bills **more than the KV pool**: hybrid models add a fixed recurrent state per slot (~150 MiB/slot for qwen35's 48 DeltaNet layers). Pool math alone under-prices.
|
||
- **The projector's `estimated worst-case memory usage of mmproj is X MiB` is NOT inside the fit projection.** It lands after the fit check, so real headroom = paper headroom − X. This is the "boots fine, then dies / dark" class.
|
||
- q8_0 K+V bills exactly `2 × (layers_kv × heads × dim / 32 blocks × 34 B)`/token — check the reported KV MiB against that to prove the dtype actually took.
|
||
|
||
## 4. Probe traps
|
||
|
||
- **First request after boot is warm-up garbage**: it paid 2.6 t/s prefill / 37.7 t/s decode where the warm run gave 1,258 t/s. Never bench the first request; discard it or warm with a throwaway.
|
||
- Reasoning-on chat probes need **≥512 `max_tokens`**: at 64 the answer was empty (`finish=length` mid-`reasoning_content`) while vision actually worked. Judge `content` only at a budget the thinking can fit.
|
||
- Raw `/completion` (no chat template) returns `content` only — thinking is invisible; `timings.prompt_n` / `predicted_per_second` are the real keys, not `prompt_tokens`.
|
||
- Vision probe: solid-color PNG ≥256 px, expect an exact color word.
|
||
- Concurrency honesty: fire two identical requests in parallel; per-slot rates from each response's `timings`.
|
||
|
||
## 5. Restore proof
|
||
|
||
`health=200` **and** `systemctl is-active` — an `inactive` unit behind a 200 means
|
||
a stray trial process holds the port. Kill by pid; never `pkill -f llama-server`.
|