agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,51 @@
|
||||
---
|
||||
name: llama-bench-fleet-sweep
|
||||
description: "Run a llama-bench sweep (batch/ubatch, ncmoe, split, or any dimension) on a Dank-fleet LLM host correctly: stop the service, run from a transient unit, one invocation per cell, sample VRAM, restore and verify — including the llama-bench vs llama-server traps that have already produced wrong numbers here."
|
||||
---
|
||||
|
||||
# llama-bench sweeps on a Dank-fleet LLM host
|
||||
|
||||
Use when tuning a llama.cpp service's serve-time parameters (batch/ubatch, `-ncmoe`, `-sm`/`-ts`, KV type) offline, i.e. any question of the form "how large may X be / what does X buy".
|
||||
|
||||
Reference implementation: `scripts/bench-batch-ubatch.sh` (repo) — read it before writing another harness.
|
||||
|
||||
## Preconditions
|
||||
|
||||
1. **Stop the service** — Benchmarking Policy in AGENTS.md is mandatory: a bench beside a live service starves the same GPU and corrupts results. Never bench against the live port.
|
||||
2. **Run from a transient unit, not a shell:**
|
||||
|
||||
```bash
|
||||
sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
|
||||
--property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
/home/dank/serverAppConfigs/scripts/<harness>.sh
|
||||
```
|
||||
|
||||
The deploy's unit sets `LimitMEMLOCK=infinity` for the ROCm staging buffers; an SSH shell gets 8 MiB (`ulimit -l`), which fails the load for a reason unrelated to what you are measuring. Passwordless sudo is available on both hosts.
|
||||
|
||||
## The traps (each one has produced a wrong number)
|
||||
|
||||
- **`/` vs `,`.** llama-bench splits `/` = devices *within one configuration*, `,` = *sweep values*. A legitimate llama-server `--tensor-split 36,29` becomes `-ts 36/29`; the comma form silently benches each card alone. Same for `--device ROCm0,ROCm1` → `-dev ROCm0/ROCm1`.
|
||||
- **No `-c`.** `n_ctx = n_prompt + n_gen + n_depth`. The context is a consequence of the test shape, so a bench can NEVER prove that a configuration's activation buffers fit beside the *deployed* KV — that requires starting the real service at the real context and slot count.
|
||||
- **A test that cannot allocate calls `exit(1)`**, killing the whole process. One invocation per cell, or one infeasible value loses every later cell — and infeasibility is usually what you are looking for.
|
||||
- **No `--spec-*` and no `--parallel`.** Decode figures are the *drafterless* baseline, and slots are never exercised. `-d <n>` is what puts KV in front of the measurement: a `tg` instance runs with `n_prompt = 0`, i.e. at ~zero depth otherwise.
|
||||
- **A cell that looks fast because its allocation was smaller than intended is indistinguishable from a fast cell** unless you sample VRAM. Sample `rocm-smi --showmeminfo vram --csv` every 2 s for the cell's whole life and report min-free per card.
|
||||
- **`local a=$1 b=$x` expands `$x` before the builtin assigns anything** — under `set -u` that is `b: unbound variable`, and it fires *after* your harness has already stopped the production service. Declare in two statements.
|
||||
- **Ranges exist:** `-ub 512-2048+256` == `512,768,...,2048`.
|
||||
|
||||
## Shape of a good harness
|
||||
|
||||
- One `llama-bench` invocation per cell; append one TSV row per cell as it completes, so progress is readable mid-run and an interrupted sweep resumes (skip cells whose `bench.json` is non-empty).
|
||||
- Write `bench.json` per cell (`-o json`), plus `bench.log` (`--progress`), plus the VRAM trace.
|
||||
- **Stop the service in the harness and restore it from a `trap ... EXIT`** — and still verify the service afterwards. A harness that crashes mid-run leaves production down.
|
||||
- Match the deployment's load flags exactly (`-ngl`, `-sm`, `-ts`, `-fa`, `-ctk/-ctv`, `-t`), or you are benching a different service. Note `-ts`/`-dev` take slashes here while the unit takes commas.
|
||||
|
||||
## Verify before burning hours
|
||||
|
||||
Dry-run the harness against a **stub** `llama-bench` (`BIN=/tmp/stubdir`, plus `SERVICE=<nonexistent-name>` so the real service is untouched) with a tiny grid, including one cell forced to `exit 3`: prove the TSV row, the failure capture, and the resume logic before the real sweep. Then validate the real flag set in ~1 s by pointing `-m` at a nonexistent file — argument errors surface *before* the model load.
|
||||
|
||||
## Interpreting results
|
||||
|
||||
- A prefill difference smaller than the run-to-run spread is not a result; check each cell's own `stddev_ts`.
|
||||
- Single-sequence decode is usually flat across batch/ubatch — that knob is prefill-side.
|
||||
- Batch ≥ ubatch always (llama.cpp clamps ubatch to batch, so smaller-batch cells silently duplicate).
|
||||
- Report what the harness cannot see (drafter, slots, real context) rather than implying the sweep settled it.
|
||||
Reference in new issue
Block a user