chore: track live agent config (config, models, append-system, managed-skills, memories, plugin lock)

2026-08-28 07:56:47 -07:00
parent 772607434a
commit 552d1ef8bf
23 changed files with 640 additions and 0 deletions
@@ -0,0 +1,69 @@
+---
+name: bench-llama-server-params
+description: "Benchmark llama.cpp server-side parameters (--threads-batch, --ubatch-size, --spec-type) using HTTP API with max_tokens=1 for prefill, max_tokens=16384+ for decode"
+---
+
+# Benchmarking llama.cpp Server-Side Parameters
+
+## Problem
+`llama-bench` benchmarks the bench binary itself, not the running server.
+It cannot measure the effect of server-side parameters like `--threads-batch`,
+`--ubatch-size`, `--spec-type`, or `--parallel`.
+
+## Solution: HTTP-Based Benchmarking
+
+### Prefill throughput (prompt processing)
+1. Start the server with the target parameter value
+2. Send a large prompt via the chat API with `max_tokens=1`
+3. Measure `prompt_tokens / wall_time` from the response usage
+
+`max_tokens=1` is correct here — you want only prompt processing time,
+not generation. The server crashes on `max_tokens=0`.
+
+### Decode throughput (generation)
+1. Start the server with the target parameter value
+2. Send a prompt that generates a long response
+3. Use `max_tokens=16384` minimum (16k) — **never `max_tokens=1`** for decode
+   benchmarking. One token tells you nothing about sustained generation rate.
+4. Measure `predicted_per_second` from the response `timings` object, or
+   compute `completion_tokens / wall_time`
+
+### Why at least 16k tokens for decode
+- Decode throughput varies over the generation (warmup, cache growth, MoE
+  routing). A single token is noise.
+- Speculative decoding acceptance rates stabilize over thousands of tokens.
+- Short generations finish before the draft head reaches steady state.
+- 16k tokens gives a stable average at ~22 tok/s (≈12 min per run).
+
+### Key Pattern
+```python
+# Decode benchmark — measure generation throughput
+body = json.dumps({
+    "model": "<alias>",
+    "messages": [{"role": "user", "content": "<prompt that generates 10k+ tokens>"}],
+    "max_tokens": 16384,   # 16k minimum for decode; use max_tokens=1 ONLY for prefill
+    "temperature": 0,     # deterministic for reproducibility
+    "stream": False
+})
+# Measure wall time from request send to response receipt
+# Get predicted_per_second from timings, or completion_tokens / wall_time
+```
+
+### When to Use
+- `--threads-batch` (prefill): use `max_tokens=1`
+- `--ubatch-size` (decode): use `max_tokens=16384+`
+- `--spec-type` / speculative decoding: use `max_tokens=16384+` (decode only)
+- Any server-side parameter that `llama-bench` cannot target
+
+### Benchmarking policy (MANDATORY)
+1. **STOP the production service** before running any benchmark
+2. **Mask** the service (`systemctl mask`, never `disable` — on these hosts
+   `disable` deletes the unit file from `/etc/systemd/system/`)
+3. Run the bench
+4. **Unmask + restart** the service when done
+5. **Verify health** before reporting results
+
+### Example Scripts
+- `scripts/bench-threads-batch-http.py` — prefill sweep (max_tokens=1)
+- `scripts/bench-decode-threads-batch.py` — decode sweep (max_tokens=256+)
+- `scripts/bench-spec-decode.py` — speculative decoding config sweep