agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,109 @@
|
||||
---
|
||||
name: llama-bench-host-sweep
|
||||
description: "Run a llama-bench sweep (batch/ubatch, split, quant, depth) on a live fleet LLM host without fooling yourself: what llama-bench cannot express, the separator trap, per-cell isolation, the memlock/transient-unit requirement, and the dry-run that must precede a multi-hour run."
|
||||
---
|
||||
|
||||
# Running a llama-bench sweep on a fleet host
|
||||
|
||||
Use when a bench must answer a serving question on one of the LLM hosts (Dankclaw
|
||||
frank-large/frank-small, Dankserver frank-medium) — batch/ubatch sizing, split
|
||||
choice, KV type, depth behaviour.
|
||||
|
||||
Repo policy applies: **stop the production service first** (AGENTS.md Benchmarking
|
||||
Policy). The harness below does that for you and restores it.
|
||||
|
||||
## 1. What llama-bench cannot express — decide this BEFORE designing the sweep
|
||||
|
||||
| Not available | Consequence |
|
||||
|---|---|
|
||||
| `-c/--ctx-size` | `n_ctx = n_prompt + n_gen + n_depth` (llama-bench.cpp). The context is a consequence of the test shape, so **the deployment's real allocation is never tested** (e.g. a 16 GiB KV at 262144 over 2 slots). Only starting the real unit answers "does it fit". |
|
||||
| `--parallel` | single sequence only; no slot shape, no concurrency effect. |
|
||||
| `--spec-*` | no drafter. Any `tg` number is the **autoregressive baseline**, not the served rate. Useful as a control, never as the deployed figure. |
|
||||
| warmup per config | a warmup run is itself a full-size run; at 64k prompts `--no-warmup` buys a second sample for the same wall clock. |
|
||||
|
||||
`-d N` is a depth PREFIX: it puts N tokens of KV in place before the measured run and
|
||||
is cached across reps through a `llama_state_seq_*` save/restore, so the fill is paid
|
||||
once per cell and is not part of the measured number. Without `-d`, a `tg` test runs at
|
||||
~zero depth and the number is meaningless for long conversations.
|
||||
|
||||
## 2. Two traps that silently change the experiment
|
||||
|
||||
- **Separators.** `/` separates devices *within one configuration*; `,` separates
|
||||
configurations to sweep. `-ts` and `-dev` both follow this. `-ts 36,29` parses as the
|
||||
single value `36` → everything lands on one card and you have benched a different
|
||||
machine. llama-server takes the comma form for the same pair. List-valued args also
|
||||
accept ranges: `-ub 512-2048+256`.
|
||||
- **memlock.** The deploy units set `LimitMEMLOCK=infinity` because the ROCm backend
|
||||
pins staging buffers; a plain SSH shell on these hosts has soft=hard=8 MiB
|
||||
(`ulimit -l`), so a load can fail for a reason unrelated to the model. Run the sweep
|
||||
inside a transient unit:
|
||||
|
||||
```bash
|
||||
sudo systemd-run --unit=fm-bench --collect --no-block \
|
||||
--property=Type=oneshot --property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
/home/dank/serverAppConfigs/scripts/bench-batch-ubatch.sh
|
||||
```
|
||||
|
||||
This also survives an ssh disconnect, unlike a foreground remote process.
|
||||
|
||||
## 3. One invocation per cell
|
||||
|
||||
A test that cannot run calls `exit(1)` immediately (failed depth/prompt/gen run), so a
|
||||
single cross-product invocation loses **every later cell** the moment one ubatch fails to
|
||||
allocate — and infeasibility is usually one of the questions being asked.
|
||||
|
||||
Drive the grid from a shell loop: one llama-bench call per (b, ub), continue on a
|
||||
non-zero exit, record why, and skip cells whose output file already exists and is
|
||||
non-empty (that makes an interrupted sweep resumable). Write rows to a TSV as the sweep
|
||||
proceeds so progress is readable mid-run.
|
||||
|
||||
## 4. Sample VRAM per cell, covering the load
|
||||
|
||||
Batch/ubatch sizes the activation buffers, so a cell that looks fast because its
|
||||
allocation was silently smaller must be distinguishable from one that is genuinely fast.
|
||||
Poll `rocm-smi --showmeminfo vram --csv` every 2 s for the cell's whole life and keep the
|
||||
peak (minimum free) per card. Correlate by taking the window between the cell's start and
|
||||
end.
|
||||
|
||||
## 5. Always restore the service from a trap
|
||||
|
||||
`trap <restart> EXIT` in the harness, plus a manual check afterwards. A harness that
|
||||
stops a production unit and then dies leaves inference down; this fleet has had that
|
||||
happen.
|
||||
|
||||
## 6. Dry-run before a multi-hour run
|
||||
|
||||
Stub the binary so the whole path is exercised without loading a 29 GiB model:
|
||||
|
||||
```bash
|
||||
# /tmp/benchstub/llama-bench prints the -o json shape and exits non-zero for one ubatch
|
||||
SERVICE=frank-medium-dryrun BIN=/tmp/benchstub MODEL=/tmp/benchstub/m.gguf OUT=/tmp/bub-test \
|
||||
N_PROMPT=8 N_GEN=4 N_DEPTH=0 REPS=1 BATCHES="2048 4096" UBATCHES="512 1024" \
|
||||
bash scripts/bench-batch-ubatch.sh
|
||||
```
|
||||
|
||||
`SERVICE=<nonexistent>` means the trap and the stop are never reached, so the real
|
||||
service is untouched. Check that a FAILED cell still produces a row with the right column
|
||||
count before trusting the table.
|
||||
|
||||
## 7. Read the result sanely
|
||||
|
||||
- Bracket the sweep: run the reference cell (the deployed configuration) first and
|
||||
again last, so drift over hours is visible.
|
||||
- Sanity-check one number against the runbook's prior figures before believing the grid.
|
||||
- Remember what the grid does NOT prove: it did not run the drafter, the slots, or the
|
||||
deployment's context. The winner still has to be applied and verified live.
|
||||
|
||||
## Pitfall while writing such a harness
|
||||
|
||||
`local b=$1 ub=$2 dir="$OUT/b${b}_ub${ub}"` expands **all** the builtin's arguments
|
||||
before the builtin assigns any of them, so under `set -u` this aborts with
|
||||
`b: unbound variable`. Declare in two statements:
|
||||
|
||||
```bash
|
||||
local b=$1 ub=$2
|
||||
local dir="$OUT/b${b}_ub${ub}" sampler start rc wall
|
||||
```
|
||||
|
||||
Ref: `scripts/bench-batch-ubatch.sh` in serverAppConfigs (the fleet's implementation of
|
||||
all of the above).
|
||||
Reference in new issue
Block a user