48 lines
4.1 KiB
Markdown
48 lines
4.1 KiB
Markdown
---
|
||
name: llamacpp-candidate-arm-census
|
||
description: "Measure a candidate GGUF/KV-shape on a live dank-fleet llama.cpp endpoint (frank-zealot pattern) in ONE service-stop window: pre-flight flag check, per-arm env overrides, boot ledger + nvidia-smi census, behavior probes, trap-guarded restore."
|
||
---
|
||
|
||
Measure candidate shapes on a production llama.cpp endpoint without cutting anything over. One outage window, many arms, guaranteed restore.
|
||
|
||
## 0. Pre-flight (no GPU needed — do all of it before stopping anything)
|
||
|
||
- `ssh <host> <candidate-bin> --list-devices` — the candidate must see the card (backend `.so` releases resolve via `$ORIGIN`, so run from the extracted asset dir).
|
||
- Flag-compat gate: loop the exact long options the serve script emits and grep the candidate's `--help`. A single unknown flag = a dead boot in the middle of the window. Remote shells may be **zsh**: `$flag[ ,<]` is parsed as an array subscript ("bad math expression") — pipe the check through `ssh <host> 'bash -s' <<'CHK' … CHK` and use `grep -qE "(^|[[:space:]])$f([[:space:],<]|$)"`.
|
||
- Pause any concurrent compile (`pkill -f "[r]un-<build>"` — the bracket stops `pkill -f` matching your own command line; `pkill -x ninja`). Ninja is incremental: the build resumes where it stopped, and CPU contention otherwise skews every rate.
|
||
|
||
## 1. Runner shape
|
||
|
||
Write one script to `/tmp` on the target via `ssh <host> 'cat >/tmp/….sh' <<'EOF'`. Never edit the repo checkout on the target; drive the shipped serve script with a **per-arm env file** and `FRANK_<SVC>_ENV_FILE` (the serve scripts already honor it).
|
||
|
||
```bash
|
||
SRV=""
|
||
used(){ nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits; }
|
||
restore(){ [[ -n "$SRV" ]] && kill "$SRV" 2>/dev/null; pkill -x llama-server
|
||
for i in $(seq 1 90); do [[ "$(used)" -lt 1500 ]] && break; sleep 2; done
|
||
sudo systemctl start <svc>
|
||
for i in $(seq 1 150); do [[ "$(curl -s -o /dev/null -w %{http_code} http://127.0.0.1:<port>/health)" == 200 ]] && break; sleep 2; done
|
||
echo "RESTORE health=$(curl -s -o /dev/null -w %{http_code} …/health) used=$(used) state=$(systemctl is-active <svc>)"; }
|
||
trap restore EXIT
|
||
```
|
||
|
||
- `trap restore EXIT` is mandatory, not polite: it also fires on the failure paths, so a script bug cannot leave the endpoint down.
|
||
- Preserve the production log first (`cp -f /home/dank/logs/<svc>.log …prod-<ts>.log`) — the serve script truncates it.
|
||
- Health-poll cap ~120×2 s. 300 polls turns an expected-allocation-failure arm (the "wall" arms) into 10 wasted minutes.
|
||
- Arm = `sed` the base env (CTX_SIZE / PARALLEL / KV_CACHE_TYPE / LLAMA_BIN / LOG_FILE), boot with `setsid nohup env FRANK_<SVC>_ENV_FILE=… bash <serve-script>`, then capture: `nvidia-smi` used+free, ledger grep (`type |model size|n_ctx|flash|KV buffer|buffer size|slots|error|exceed`), `/slots` (per-slot `n_ctx` is the harness contract), `/v1/models`.
|
||
- **Point LLAMA_BIN at a binary that exists.** Take 1 of this procedure died in 8 s because the trial env named a still-compiling source build; the trap saved the endpoint.
|
||
|
||
## 2. Probes that actually discriminate
|
||
|
||
- chat: `content` vs `reasoning_content` (channel differs per runtime) + `finish_reason`.
|
||
- tools: real `tools` schema, assert `tool_calls[0].function.arguments` parses as JSON — empty/`null` arguments is a known fork bug.
|
||
- vision: 512×512 **solid color** PNG built with `zlib`+`struct` (no PIL), sent as an OpenAI `image_url` content part; expect the exact color word. The legacy top-level `images` field hallucinates a plausible answer.
|
||
- rate: `/completion` (not `/v1/chat/completions`) — it returns `timings` (prefill/decode). ≥128 tokens.
|
||
- long-context fill: ~190k tokens through `/completion` proves the per-slot window AND tests the "quantized KV near 200K" stability reports.
|
||
|
||
Skip `llama-bench` for shape work: it has no `-c`/`-np`, and its `-mmp` is `--mmap` (deprecated), not mmproj. Booting the server per arm measures the real shape and prints the ledger, which is the referee.
|
||
|
||
## 3. Report
|
||
|
||
Per arm: ledger KV-buffer MiB, used/free at settle, FITS / tight / NO FIT vs the card's `used+free` ceiling from settled boots (not the spec sheet). Then restore, and state the downtime.
|