Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
71 lines
3.4 KiB
Markdown
71 lines
3.4 KiB
Markdown
---
|
|
name: live-llm-benchmark-hygiene
|
|
description: "Measure and tune a live LLM server without fooling yourself: workload-matched corpora, salted runs against warm prefix caches, turn-level clients, engine ledger for capacity, soak tests, and reboot verification."
|
|
---
|
|
|
|
# Benchmarking a live LLM server honestly
|
|
|
|
Use when measuring or tuning a served model (vLLM, llama.cpp, halogen, radiance,
|
|
any OpenAI-compatible endpoint). The point is to avoid the four confounds that
|
|
turned a night's numbers into fiction before each one was fixed.
|
|
|
|
## 1. Match the corpus to the workload
|
|
|
|
Synthetic token soup (`vllm bench serve --dataset-name random`) under-reports
|
|
speculative decoding badly: measured 2.9-3.6 tok/update against 3.86 on real
|
|
code, and acceptance — not step time — is what moves throughput. For a coding
|
|
fleet, build the corpus from the repo's own sources: file body plus an
|
|
instruction that requires reading it. Then `tok/update` and TPOT mean something.
|
|
|
|
## 2. Salt every run so the prefix cache cannot help
|
|
|
|
vLLM runs with `--enable-prefix-caching`, and a repeated corpus silently
|
|
measures a warm cache. Put a unique per-run token in the first bytes (it is both
|
|
the engine's cache key and, if a proxy routes on it, the routing key). Without
|
|
this, the second run of any comparison is not comparable to the first.
|
|
|
|
## 3. Measure turns, not only batches
|
|
|
|
A batch of distinct prompts parallelises prefill and hides the real cost of a
|
|
conversation: an agent loop re-sends its context every turn. Two workloads need
|
|
two clients:
|
|
|
|
- **batch/throughput** — `vllm bench serve --dataset-name custom --dataset-path
|
|
<jsonl>` with `--custom-output-len`, several concurrency levels.
|
|
- **sequential turns** — a small client that sends turn after turn with growing
|
|
history and a large shared prefix, timing **TTFT of the first streamed chunk**
|
|
per turn. This is the only measurement that exposes prefix-cache behaviour.
|
|
|
|
## 4. Read the engine's own ledger for capacity
|
|
|
|
Do not infer memory from the card's size. The server prints the truth at boot:
|
|
|
|
```
|
|
Available KV cache memory: 5.44 GiB
|
|
GPU KV cache size: 122,636 tokens
|
|
Maximum concurrency for 114,688 tokens per request: 1.07x
|
|
```
|
|
|
|
That is the number to quote for "how many concurrent conversations fit".
|
|
Related trap: the arena/KV budget is **not monotonic in chunk size** — on one
|
|
host, available KV was 5.44 GiB at `CHUNK=8192`, 4.22 GiB at 16384 and 4.16 GiB
|
|
at 4096 (where it would not even boot). Never assume "smaller chunk frees KV";
|
|
measure three points before treating a knob as a lever.
|
|
|
|
## 5. Soak before believing a tuning change
|
|
|
|
Run several families concurrently for 15-30 minutes and count failures, not just
|
|
latency: a nightly soak returned 155/155 answers with zero failovers while only
|
|
25% of turns hit the cache — which is how the capacity ceiling was found as a
|
|
*latency* effect rather than an error. Report hit/miss split, medians, and the
|
|
min/max, because the median of a bimodal distribution describes neither mode.
|
|
|
|
## 6. Verification that is not the benchmark
|
|
|
|
- Run the client the fleet actually uses (e.g. `omp --model <alias> -p
|
|
--no-tools`) — a config file that parses is not a config that works, and a
|
|
non-obvious API mode (raw `/v1/completions`) may still be fine because the
|
|
client templates its own prompt.
|
|
- A deployed inference unit is not proven until an **unattended reboot** brings
|
|
it back: verify `is-active`, `/health`, and one real completion afterwards.
|